[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:12 dist-test-slave-f4rg krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:12 dist-test-slave-f4rg krb5kdc[399](info): set up 2 sockets
Sep 27 02:47:12 dist-test-slave-f4rg krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 02:47:13.083565 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:13.197492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38017
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:13.276351 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:13.276574 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:13.276597 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:13.277756 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:13.277802 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:13.277817 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:13.277827 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:13.277839 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:13.279627 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:13.280018 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:13.280323 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:47:13.285804 412 server_base.cc:1034] running on GCE node
W20260927 02:47:13.285883 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:13.285887 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:13.286132 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:13.286367 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:13.286645 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:13.287782 412 hybrid_clock.cc:648] HybridClock initialized: now 1790477233287752 us; error 49 us; skew 500 ppm
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:13.290994 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:13.291543 412 webserver.cc:460] Webserver started at http://127.0.98.62:41261/ using document root <none> and password file <none>
I20260927 02:47:13.291757 412 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:13.291802 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:13.291962 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:13.293008 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "6399d1ea0e224b90bb78f25140cd10be"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "ad843fb94468f44f0249e42be931a748"
server_key_iv: "cf4394a9cd7aacb4166ccfd35a8c3422"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.293370 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "6399d1ea0e224b90bb78f25140cd10be"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "ad843fb94468f44f0249e42be931a748"
server_key_iv: "cf4394a9cd7aacb4166ccfd35a8c3422"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.295091 412 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 02:47:13.295917 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.296082 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:13.296164 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "6399d1ea0e224b90bb78f25140cd10be"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "ad843fb94468f44f0249e42be931a748"
server_key_iv: "cf4394a9cd7aacb4166ccfd35a8c3422"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.296236 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:13.317672 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:13.319494 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:13.319638 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:13.323366 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38017
I20260927 02:47:13.323405 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38017 every 8 connection(s)
I20260927 02:47:13.323740 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:13.324585 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:13.326784 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be: Bootstrap starting.
I20260927 02:47:13.327355 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:13.327570 480 log.cc:824] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:13.328168 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be: No bootstrap required, opened a new log
I20260927 02:47:13.329730 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6399d1ea0e224b90bb78f25140cd10be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38017 } }
I20260927 02:47:13.329907 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:13.329931 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6399d1ea0e224b90bb78f25140cd10be, State: Initialized, Role: FOLLOWER
I20260927 02:47:13.330045 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6399d1ea0e224b90bb78f25140cd10be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38017 } }
I20260927 02:47:13.330120 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:13.330152 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:13.330183 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:13.330735 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6399d1ea0e224b90bb78f25140cd10be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38017 } }
I20260927 02:47:13.330854 480 leader_election.cc:302] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6399d1ea0e224b90bb78f25140cd10be; no voters:
I20260927 02:47:13.331019 480 leader_election.cc:288] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:13.331080 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:13.331229 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [term 1 LEADER]: Becoming Leader. State: Replica: 6399d1ea0e224b90bb78f25140cd10be, State: Running, Role: LEADER
I20260927 02:47:13.331241 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:13.331331 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6399d1ea0e224b90bb78f25140cd10be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38017 } }
I20260927 02:47:13.331858 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6399d1ea0e224b90bb78f25140cd10be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6399d1ea0e224b90bb78f25140cd10be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38017 } } }
I20260927 02:47:13.331998 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:13.332068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 412
I20260927 02:47:13.332192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:13.332288 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6399d1ea0e224b90bb78f25140cd10be. Latest consensus state: current_term: 1 leader_uuid: "6399d1ea0e224b90bb78f25140cd10be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6399d1ea0e224b90bb78f25140cd10be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38017 } } }
I20260927 02:47:13.332396 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:13.332677 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:13.332971 392 external_mini_cluster.cc:1428] Setting key 87ae15936e42de652863ce01c31b8d62
I20260927 02:47:13.333222 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:13.334676 492 catalog_manager.cc:1347] Generated new cluster ID: 2fb7a1c5263b4a6e90810d03b5246221
I20260927 02:47:13.334721 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:13.350499 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.333852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.98.62:38017)
0927 02:47:13.334022 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:13.334031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:13.334346 (+ 315us) server_negotiation.cc:407] Connection header received
0927 02:47:13.336604 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:13.336611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:13.336662 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:13.336793 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:13.338775 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.340179 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.341638 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.342227 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.342316 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.346393 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:13.346397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:13.346397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:13.346410 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:13.348564 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.349024 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.349039 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.349040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.349151 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.349387 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.349388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.349388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.349720 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:13.349857 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:13.350113 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 02:47:13.350257 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260927 02:47:13.361570 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:13.362326 492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:13.366640 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6399d1ea0e224b90bb78f25140cd10be: Generated new TSK 0
I20260927 02:47:13.366852 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:13.393644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38017
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:13.473397 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:13.473614 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:13.473646 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:13.474773 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:13.474835 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:13.474875 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:13.476312 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:13.476559 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:13.476768 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:13.481488 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:13.481499 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:13.481551 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:13.481607 506 server_base.cc:1034] running on GCE node
I20260927 02:47:13.481874 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:13.482096 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:13.483242 506 hybrid_clock.cc:648] HybridClock initialized: now 1790477233483233 us; error 27 us; skew 500 ppm
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:13.485496 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:13.485936 506 webserver.cc:460] Webserver started at http://127.0.98.1:45691/ using document root <none> and password file <none>
I20260927 02:47:13.486141 506 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:13.486189 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:13.486320 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:13.487375 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "14e94a7b4ec343c89ef4de8339387471"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "89803e463ee5b782eee9dbe63404f749"
server_key_iv: "8a89ba2a5224625b938b66344ffeab43"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.487695 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "14e94a7b4ec343c89ef4de8339387471"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "89803e463ee5b782eee9dbe63404f749"
server_key_iv: "8a89ba2a5224625b938b66344ffeab43"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.488888 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:13.489523 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.489672 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:13.489750 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "14e94a7b4ec343c89ef4de8339387471"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "89803e463ee5b782eee9dbe63404f749"
server_key_iv: "8a89ba2a5224625b938b66344ffeab43"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.489809 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:13.499886 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:13.501601 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:13.501775 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:13.501973 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:13.502255 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:13.502286 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.502313 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:13.502337 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.507722 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34181
I20260927 02:47:13.507773 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34181 every 8 connection(s)
I20260927 02:47:13.508054 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:13.508181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 506
I20260927 02:47:13.508281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:13.508451 392 external_mini_cluster.cc:1428] Setting key a3aa146c14cf9da8c4c3f1cc1e2edd63
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:47:13.517505 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.508753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59485 (local address 127.0.98.62:38017)
0927 02:47:13.508822 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:13.508825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:13.509022 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:47:13.509563 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:13.509564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:13.509579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:13.509648 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:13.510537 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.511158 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.512253 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.512555 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.512621 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.514960 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:13.514961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:13.514962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:13.514971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:13.516366 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.516759 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.516760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.516761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.516822 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.517018 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.517019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.517019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.517123 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:13.517170 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:13.517373 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 02:47:13.517440 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:47:13.517797 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.508761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38017 (local address 127.0.98.1:59485)
0927 02:47:13.508914 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:13.508955 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:13.509448 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:13.509705 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:13.509708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:13.510249 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:13.510446 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:13.510450 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.511239 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.511240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:13.512163 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:13.512165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.512657 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.512658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:13.512772 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.513310 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:13.513318 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:13.514874 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:13.516445 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:13.516451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:13.516453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:13.516693 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:13.516883 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:13.516884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:13.516884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:13.516976 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:13.517166 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:13.517169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:13.517281 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 02:47:13.517352 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:47:13.517474 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260927 02:47:13.518278 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38017
I20260927 02:47:13.518368 635 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:13.518533 635 heartbeater.cc:511] Master 127.0.98.62:38017 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:47:13.519088 444 ts_manager.cc:194] Registered new tserver with Master: 14e94a7b4ec343c89ef4de8339387471 (127.0.98.1:34181)
I20260927 02:47:13.519696 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:13.532677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38017
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:13.611683 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:13.611901 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:13.611934 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:13.613111 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:13.613155 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:13.613191 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:13.614645 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:13.614916 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:13.615131 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:13.619884 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:13.619923 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:13.619940 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:13.619994 640 server_base.cc:1034] running on GCE node
I20260927 02:47:13.620338 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:13.620563 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:13.621701 640 hybrid_clock.cc:648] HybridClock initialized: now 1790477233621683 us; error 34 us; skew 500 ppm
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:13.624028 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:13.624434 640 webserver.cc:460] Webserver started at http://127.0.98.2:44285/ using document root <none> and password file <none>
I20260927 02:47:13.624673 640 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:13.624720 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:13.624842 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:13.625931 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "e93b67f2cd9440cf8760a02b81aec32e"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "770b8e71d245a07d31b0fda41eff96c8"
server_key_iv: "2f9ceac23ec7c78fd1937320ca8e5e92"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.626256 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e93b67f2cd9440cf8760a02b81aec32e"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "770b8e71d245a07d31b0fda41eff96c8"
server_key_iv: "2f9ceac23ec7c78fd1937320ca8e5e92"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.627566 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:13.628240 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.628387 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.628454 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "e93b67f2cd9440cf8760a02b81aec32e"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "770b8e71d245a07d31b0fda41eff96c8"
server_key_iv: "2f9ceac23ec7c78fd1937320ca8e5e92"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.628518 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:13.634896 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:13.636543 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:13.636673 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:13.636866 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:13.637125 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:13.637157 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.637187 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:13.637210 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.643208 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45599
I20260927 02:47:13.643257 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45599 every 8 connection(s)
I20260927 02:47:13.643572 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:13.646975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 640
I20260927 02:47:13.647066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:13.647208 392 external_mini_cluster.cc:1428] Setting key 5d21a45bf86f8a571b9ad78e34d5bce2
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:13.652915 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.644242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36487 (local address 127.0.98.62:38017)
0927 02:47:13.644297 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:13.644299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:13.644469 (+ 170us) server_negotiation.cc:407] Connection header received
0927 02:47:13.645021 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:13.645022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:13.645035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:13.645086 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:13.646007 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.646511 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.647638 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.647838 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.647893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.650075 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:13.650076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:13.650077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:13.650084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:13.651579 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.651983 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.651984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.651984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.652054 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.652265 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.652266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.652266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.652408 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:13.652487 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:13.652742 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 02:47:13.652821 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:47:13.652967 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.644217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38017 (local address 127.0.98.2:36487)
0927 02:47:13.644357 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:13.644393 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:13.644926 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:13.645154 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:13.645157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:13.645722 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:13.645918 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:13.645922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.646599 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.646600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:13.647564 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:13.647566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.647906 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.647907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:13.648031 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.648518 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:13.648527 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:13.649984 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:13.651684 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:13.651690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:13.651692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:13.651913 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:13.652110 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:13.652111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:13.652111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:13.652200 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:13.652474 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:13.652477 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:13.652596 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 02:47:13.652672 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:47:13.652797 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260927 02:47:13.653426 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38017
I20260927 02:47:13.653533 769 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:13.653738 769 heartbeater.cc:511] Master 127.0.98.62:38017 requested a full tablet report, sending...
I20260927 02:47:13.654119 444 ts_manager.cc:194] Registered new tserver with Master: e93b67f2cd9440cf8760a02b81aec32e (127.0.98.2:45599)
I20260927 02:47:13.654490 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36487
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:13.671411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38017
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:13.750439 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:13.750646 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:13.750669 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:13.751775 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:13.751811 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:13.751835 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:13.753206 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:13.753391 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:13.753566 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:13.758196 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:13.758230 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:13.758355 774 server_base.cc:1034] running on GCE node
W20260927 02:47:13.758231 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:13.758608 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:13.758876 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:13.760023 774 hybrid_clock.cc:648] HybridClock initialized: now 1790477233760010 us; error 30 us; skew 500 ppm
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:13.762320 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:13.762778 774 webserver.cc:460] Webserver started at http://127.0.98.3:39505/ using document root <none> and password file <none>
I20260927 02:47:13.763010 774 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:13.763056 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:13.763180 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:13.764170 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ee0202ed744450b9b9f27549c8a0fb5"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "240115180ad7bf469cdf40de58951474"
server_key_iv: "95479767da2ebf8b911379050b8a6c82"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.764487 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ee0202ed744450b9b9f27549c8a0fb5"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "240115180ad7bf469cdf40de58951474"
server_key_iv: "95479767da2ebf8b911379050b8a6c82"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.765733 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:13.766317 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.766466 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:13.766543 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "4ee0202ed744450b9b9f27549c8a0fb5"
format_stamp: "Formatted at 2026-09-27 02:47:13 on dist-test-slave-f4rg"
server_key: "240115180ad7bf469cdf40de58951474"
server_key_iv: "95479767da2ebf8b911379050b8a6c82"
server_key_version: "encryptionkey@0"
I20260927 02:47:13.766598 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:13.788566 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:13.790311 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:13.790449 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:13.790668 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:13.790983 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:13.791016 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.791046 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:13.791069 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:13.797402 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46537
I20260927 02:47:13.797461 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46537 every 8 connection(s)
I20260927 02:47:13.797757 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:13.805789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 774
I20260927 02:47:13.805889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:13.806028 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.798374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47585 (local address 127.0.98.62:38017)
0927 02:47:13.798417 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:13.798419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:13.798651 (+ 232us) server_negotiation.cc:407] Connection header received
0927 02:47:13.799168 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:13.799169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:13.799180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:13.799228 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:13.800165 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.800646 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.801608 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.801792 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.801847 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.803904 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:13.803905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:13.803906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:13.803912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:13.804944 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.805279 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.805280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.805280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.805324 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.805497 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.805497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.805498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.805592 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:13.805690 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:13.805866 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:47:13.805941 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:47:13.806052 392 external_mini_cluster.cc:1428] Setting key 0e2b3f3220fd956cb6f56af472bf3e5e
I20260927 02:47:13.806265 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.798371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38017 (local address 127.0.98.3:47585)
0927 02:47:13.798524 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:13.798559 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:13.799076 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:13.799307 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:13.799311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:13.799877 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:13.800068 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:13.800072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.800725 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.800726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:13.801524 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:13.801526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.801859 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.801860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:13.801978 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.802413 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:13.802421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:13.803811 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:13.805005 (+ 1194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:13.805009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:13.805011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:13.805214 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:13.805367 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:13.805367 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:13.805368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:13.805457 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:13.805634 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:13.805637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:13.805749 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 02:47:13.805827 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:47:13.805983 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260927 02:47:13.806691 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38017
I20260927 02:47:13.806852 903 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:13.807045 903 heartbeater.cc:511] Master 127.0.98.62:38017 requested a full tablet report, sending...
I20260927 02:47:13.807402 444 ts_manager.cc:194] Registered new tserver with Master: 4ee0202ed744450b9b9f27549c8a0fb5 (127.0.98.3:46537)
I20260927 02:47:13.807789 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47585
I20260927 02:47:13.818176 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:13.840211 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.833113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52980 (local address 127.0.98.1:34181)
0927 02:47:13.833265 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:13.833270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:13.833281 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:47:13.833303 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:13.833305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:13.833318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:13.833381 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:13.833948 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.834657 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.835410 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.835721 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.835779 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.837820 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:13.837828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:13.837828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:13.837836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:13.839178 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.839469 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.839472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.839473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.839522 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.839686 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.839687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.839687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.839816 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:13.839946 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:13.840011 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:47:13.840082 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260927 02:47:13.840547 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52980
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:13.860682 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:13.854416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.0.98.1:34181)
0927 02:47:13.854483 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:13.854485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:13.854521 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:47:13.854587 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:13.854588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:13.854601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:13.854649 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:13.855172 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.855672 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.856343 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:13.856539 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:13.856588 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:13.858574 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:13.858575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:13.858575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:13.858583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:13.859845 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.860157 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.860158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.860158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.860201 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:13.860332 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:13.860332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:13.860332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:13.860416 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:13.860495 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:13.860551 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:47:13.860608 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 02:47:13.861186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 506
I20260927 02:47:13.865967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 640
I20260927 02:47:13.870777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 774
I20260927 02:47:13.874708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 412
2026-09-27T02:47:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1740 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[917](info): set up 2 sockets
Sep 27 02:47:13 dist-test-slave-f4rg krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 02:47:14.818177 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:14 dist-test-slave-f4rg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:14.928814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:15.006647 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:15.006896 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:15.006923 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:15.008086 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:15.008127 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:15.008140 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:15.008152 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:15.008162 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:15.009655 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:15.009883 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:15.010099 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:15.014663 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:15.014683 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:15.014770 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:15.014883 930 server_base.cc:1034] running on GCE node
I20260927 02:47:15.015125 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:15.015372 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:15.016505 930 hybrid_clock.cc:648] HybridClock initialized: now 1790477235016491 us; error 29 us; skew 500 ppm
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:15.018947 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:15.019347 930 webserver.cc:460] Webserver started at http://127.0.98.62:33597/ using document root <none> and password file <none>
I20260927 02:47:15.019553 930 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:15.019600 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:15.019717 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:15.020740 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "79cdeeb8f6994ea7aef45215a35af041"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "08071a4a1812b926a30f26245068c1d3"
server_key_iv: "34b1d47025cd6a116f18aafe5022a25b"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.021073 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "79cdeeb8f6994ea7aef45215a35af041"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "08071a4a1812b926a30f26245068c1d3"
server_key_iv: "34b1d47025cd6a116f18aafe5022a25b"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.022352 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:15.023154 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.023339 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:15.023430 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "79cdeeb8f6994ea7aef45215a35af041"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "08071a4a1812b926a30f26245068c1d3"
server_key_iv: "34b1d47025cd6a116f18aafe5022a25b"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.023494 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:15.042013 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:15.043928 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:15.044060 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:15.047564 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41107
I20260927 02:47:15.047598 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41107 every 8 connection(s)
I20260927 02:47:15.047907 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:15.048739 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:15.050933 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041: Bootstrap starting.
I20260927 02:47:15.051446 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:15.051680 998 log.cc:824] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:15.052268 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041: No bootstrap required, opened a new log
I20260927 02:47:15.053092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 930
I20260927 02:47:15.053179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:15.053328 392 external_mini_cluster.cc:1428] Setting key 222d30603238930c89250c0e7a42ebf9
I20260927 02:47:15.054177 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cdeeb8f6994ea7aef45215a35af041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41107 } }
I20260927 02:47:15.054320 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:15.054353 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cdeeb8f6994ea7aef45215a35af041, State: Initialized, Role: FOLLOWER
I20260927 02:47:15.054488 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cdeeb8f6994ea7aef45215a35af041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41107 } }
I20260927 02:47:15.054565 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:15.054612 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:15.054661 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:15.055567 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cdeeb8f6994ea7aef45215a35af041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41107 } }
I20260927 02:47:15.055661 998 leader_election.cc:302] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79cdeeb8f6994ea7aef45215a35af041; no voters:
I20260927 02:47:15.055797 998 leader_election.cc:288] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:15.055838 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:15.055975 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [term 1 LEADER]: Becoming Leader. State: Replica: 79cdeeb8f6994ea7aef45215a35af041, State: Running, Role: LEADER
I20260927 02:47:15.056044 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:15.056058 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cdeeb8f6994ea7aef45215a35af041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41107 } }
I20260927 02:47:15.056492 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79cdeeb8f6994ea7aef45215a35af041. Latest consensus state: current_term: 1 leader_uuid: "79cdeeb8f6994ea7aef45215a35af041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cdeeb8f6994ea7aef45215a35af041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41107 } } }
I20260927 02:47:15.056576 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:15.056710 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79cdeeb8f6994ea7aef45215a35af041" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79cdeeb8f6994ea7aef45215a35af041" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41107 } } }
I20260927 02:47:15.056780 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:15.056761 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:15.057668 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:15.059294 1011 catalog_manager.cc:1347] Generated new cluster ID: 30af5ef2122e4f77b5fb17b7c7f84947
I20260927 02:47:15.059346 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:15.063205 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.053828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.0.98.62:41107)
0927 02:47:15.053974 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.053982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.054006 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:47:15.054138 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.054144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.054187 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.054306 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.055012 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.056096 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.057156 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.057727 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.057854 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.060263 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.060267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.060268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.060280 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.061770 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.062083 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.062086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.062086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.062180 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.062349 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.062350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.062350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.062582 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.062730 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.062795 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.063043 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260927 02:47:15.092427 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:15.092984 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:15.099632 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79cdeeb8f6994ea7aef45215a35af041: Generated new TSK 0
I20260927 02:47:15.099828 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:15.133293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41107
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:15.212474 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:15.212677 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:15.212708 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:15.213886 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:15.213925 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:15.213961 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:15.215479 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:15.215714 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:15.215926 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:15.220821 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:15.220829 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:15.220821 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:15.221079 1024 server_base.cc:1034] running on GCE node
I20260927 02:47:15.221293 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:15.221520 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:15.222678 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790477235222659 us; error 35 us; skew 500 ppm
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:15.225261 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:15.225687 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45691/ using document root <none> and password file <none>
I20260927 02:47:15.225905 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:15.225948 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:15.226059 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:15.227177 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "e701bdce88254e86a521daaaa2d9a763"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "f349779372e551c922027c7c24d757e8"
server_key_iv: "492f9149e8d9871618a8faaad3871c64"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.227478 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e701bdce88254e86a521daaaa2d9a763"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "f349779372e551c922027c7c24d757e8"
server_key_iv: "492f9149e8d9871618a8faaad3871c64"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.228677 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:15.229331 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.229490 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:15.229578 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "e701bdce88254e86a521daaaa2d9a763"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "f349779372e551c922027c7c24d757e8"
server_key_iv: "492f9149e8d9871618a8faaad3871c64"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.229640 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:15.239997 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:15.241679 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:15.241786 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:15.241958 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:15.242202 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:15.242236 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.242256 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:15.242272 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.247596 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36153
I20260927 02:47:15.247634 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36153 every 8 connection(s)
I20260927 02:47:15.247938 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:15.256589 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.248593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37609 (local address 127.0.98.62:41107)
0927 02:47:15.248651 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.248653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.248829 (+ 176us) server_negotiation.cc:407] Connection header received
0927 02:47:15.249311 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.249313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.249325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.249367 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.250305 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.250789 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.251800 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.252004 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.252056 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.254169 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.254170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.254171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.254180 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.255344 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.255720 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.255721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.255721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.255777 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.256013 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.256014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.256014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.256119 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.256179 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.256448 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.256522 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:47:15.256778 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.248568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41107 (local address 127.0.98.1:37609)
0927 02:47:15.248697 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.248731 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.249218 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.249445 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.249449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.250013 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:15.250204 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.250209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.250911 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.250912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.251704 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.251705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.252071 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.252072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.252193 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.252687 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:15.252696 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:15.254077 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:15.255419 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:15.255424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:15.255426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:15.255625 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:15.255863 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:15.255863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:15.255864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:15.255964 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:15.256170 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:15.256174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:15.256322 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.256424 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.256567 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260927 02:47:15.257233 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41107
I20260927 02:47:15.257332 1153 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:15.257508 1153 heartbeater.cc:511] Master 127.0.98.62:41107 requested a full tablet report, sending...
I20260927 02:47:15.258003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 1024
I20260927 02:47:15.258035 962 ts_manager.cc:194] Registered new tserver with Master: e701bdce88254e86a521daaaa2d9a763 (127.0.98.1:36153)
I20260927 02:47:15.258080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:15.258246 392 external_mini_cluster.cc:1428] Setting key d9635db958cf7be3082856560efd7dc2
I20260927 02:47:15.258715 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37609
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:15.281105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41107
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:15.370790 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:15.371028 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:15.371052 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:15.372202 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:15.372246 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:15.372272 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:15.373638 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:15.373834 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:15.374011 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:15.378647 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:15.378649 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:15.378731 1158 server_base.cc:1034] running on GCE node
W20260927 02:47:15.378722 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:15.379020 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:15.379235 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:15.380366 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790477235380354 us; error 29 us; skew 500 ppm
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:15.382736 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:15.383167 1158 webserver.cc:460] Webserver started at http://127.0.98.2:45597/ using document root <none> and password file <none>
I20260927 02:47:15.383359 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:15.383405 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:15.383526 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:15.384532 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "3b6824e2caf74ab88b1a3f66b51bcf54"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "fde02de44ddedac88bce09bbc272ea59"
server_key_iv: "6010fb6d7fb706a115c3b377be236354"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.384859 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3b6824e2caf74ab88b1a3f66b51bcf54"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "fde02de44ddedac88bce09bbc272ea59"
server_key_iv: "6010fb6d7fb706a115c3b377be236354"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.386067 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:15.386667 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.386855 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:15.386936 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "3b6824e2caf74ab88b1a3f66b51bcf54"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "fde02de44ddedac88bce09bbc272ea59"
server_key_iv: "6010fb6d7fb706a115c3b377be236354"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.387004 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:15.409358 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:15.411655 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:15.411803 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:15.412017 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:15.412318 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:15.412351 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.412380 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:15.412403 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.417701 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43831
I20260927 02:47:15.417738 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43831 every 8 connection(s)
I20260927 02:47:15.418043 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:15.419683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 1158
I20260927 02:47:15.419780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:15.419945 392 external_mini_cluster.cc:1428] Setting key d7ca07ce67f4f0e2a1e42391e858c073
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:15.427191 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.418696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34679 (local address 127.0.98.62:41107)
0927 02:47:15.418745 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.418748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.418971 (+ 223us) server_negotiation.cc:407] Connection header received
0927 02:47:15.419452 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.419453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.419463 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.419500 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.420569 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.421056 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.422304 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.422496 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.422545 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.424530 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.424532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.424532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.424540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.425802 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.426150 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.426151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.426152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.426203 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.426440 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.426442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.426442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.426589 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.426666 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.426998 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.427083 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:47:15.427245 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.418680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41107 (local address 127.0.98.2:34679)
0927 02:47:15.418857 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.418894 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.419363 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.419575 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.419579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.420283 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:15.420496 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.420503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.421137 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.421138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.422209 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.422212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.422567 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.422568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.422686 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.423185 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:15.423193 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:15.424442 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:15.425865 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:15.425872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:15.425874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:15.426086 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:15.426254 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:15.426254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:15.426255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:15.426367 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:15.426655 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:15.426659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:15.426853 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.426956 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.427079 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:47:15.427692 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41107
I20260927 02:47:15.427793 1287 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:15.427960 1287 heartbeater.cc:511] Master 127.0.98.62:41107 requested a full tablet report, sending...
I20260927 02:47:15.428258 962 ts_manager.cc:194] Registered new tserver with Master: 3b6824e2caf74ab88b1a3f66b51bcf54 (127.0.98.2:43831)
I20260927 02:47:15.428606 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:15.443583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41107
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:15.523216 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:15.523422 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:15.523444 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:15.524646 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:15.524688 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:15.524714 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:15.526186 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41107
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:15.526417 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:15.526644 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:47:15.531445 1292 server_base.cc:1034] running on GCE node
W20260927 02:47:15.531416 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:15.531415 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:15.531420 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:15.531803 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:15.532034 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:15.533174 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790477235533159 us; error 28 us; skew 500 ppm
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:15.535476 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:15.535907 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37375/ using document root <none> and password file <none>
I20260927 02:47:15.536098 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:15.536137 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:15.536227 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:15.537252 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "37a2d44624d94150b946e36dcee792a8"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "6f648ba448fa3d24755a2f5c520570b2"
server_key_iv: "3b76b73e836682e6b56cded7c1b3787d"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.537577 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37a2d44624d94150b946e36dcee792a8"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "6f648ba448fa3d24755a2f5c520570b2"
server_key_iv: "3b76b73e836682e6b56cded7c1b3787d"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.538791 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:47:15.539481 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.539682 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:15.539760 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "37a2d44624d94150b946e36dcee792a8"
format_stamp: "Formatted at 2026-09-27 02:47:15 on dist-test-slave-f4rg"
server_key: "6f648ba448fa3d24755a2f5c520570b2"
server_key_iv: "3b76b73e836682e6b56cded7c1b3787d"
server_key_version: "encryptionkey@0"
I20260927 02:47:15.539812 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:15.558271 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:15.560405 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:15.560515 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:15.560694 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:15.560972 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:15.561004 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.561024 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:15.561048 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:15.566246 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36789
I20260927 02:47:15.566289 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36789 every 8 connection(s)
I20260927 02:47:15.566577 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:15.567941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 1292
I20260927 02:47:15.568022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:15.568164 392 external_mini_cluster.cc:1428] Setting key 454ea18e62d0170e5f700576782f5a98
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:15.574994 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.567234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56685 (local address 127.0.98.62:41107)
0927 02:47:15.567289 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.567291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.567462 (+ 171us) server_negotiation.cc:407] Connection header received
0927 02:47:15.567975 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.567976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.567989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.568041 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.569004 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.569499 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.570516 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.570709 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.570762 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.572690 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.572692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.572692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.572699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.573791 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.574150 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.574151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.574152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.574203 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.574400 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.574401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.574401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.574503 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.574566 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.574829 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.574907 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:47:15.575181 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.567217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41107 (local address 127.0.98.3:56685)
0927 02:47:15.567351 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.567385 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.567882 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.568117 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.568120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.568676 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:15.568924 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.568930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.569639 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.569640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.570439 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.570441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.570774 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.570775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.570913 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.571341 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:15.571348 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:15.572607 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:15.573860 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:15.573866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:15.573868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:15.574087 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:15.574253 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:15.574254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:15.574254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:15.574349 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:15.574553 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:15.574558 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:15.574691 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.574777 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.574948 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260927 02:47:15.575574 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41107
I20260927 02:47:15.575714 1421 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:15.575878 1421 heartbeater.cc:511] Master 127.0.98.62:41107 requested a full tablet report, sending...
I20260927 02:47:15.576220 962 ts_manager.cc:194] Registered new tserver with Master: 37a2d44624d94150b946e36dcee792a8 (127.0.98.3:36789)
I20260927 02:47:15.576565 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56685
I20260927 02:47:15.579638 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:47:15.586333 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.581064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.0.98.62:41107)
0927 02:47:15.581121 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.581123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.581192 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:47:15.581259 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.581260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.581270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.581312 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.581820 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.582277 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.582998 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.583174 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.583224 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.584251 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.584252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.584252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.584260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.585345 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.585630 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.585631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.585632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.585682 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.585838 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.585839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.585839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.585983 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.586067 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.586174 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.586245 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:47:15.588259 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:15.589802 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:15.595498 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.591718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36789 (local address 127.0.0.1:46888)
0927 02:47:15.592158 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.592167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.592223 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.592455 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.592456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.592457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:15.592503 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.592504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.593438 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.593439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.594321 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.594323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.595193 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.595194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.595288 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.595303 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.595371 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.595421 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":59,"threads_started":1}
I20260927 02:47:15.595515 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.591923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.0.98.3:36789)
0927 02:47:15.592078 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.592104 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.592225 (+ 121us) server_negotiation.cc:407] Connection header received
0927 02:47:15.592295 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.592297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.592314 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.592386 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:15.592572 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.593346 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.594394 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.595117 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.595193 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.595248 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.595401 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.595439 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260927 02:47:15.596040 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.591705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36153 (local address 127.0.0.1:43362)
0927 02:47:15.591968 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.592011 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.592076 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.592250 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.592253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.592255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:15.592317 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.592319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.593720 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.593722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.594895 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.594898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.595808 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.595809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.595884 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.595887 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.595933 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.595973 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260927 02:47:15.596068 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.591808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.98.1:36153)
0927 02:47:15.592106 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.592110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.592121 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:47:15.592158 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.592160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.592174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.592242 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:15.592396 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.593607 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.594965 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.595665 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.595726 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.595775 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.595958 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.595998 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":73,"threads_started":1}
I20260927 02:47:15.596329 1355 tablet_service.cc:1467] Processing CreateTablet for tablet b2d5c293d3ac4392a21282a5a230fdc4 (DEFAULT_TABLE table=test-table [id=6ec3d9c12d8b4b0c89dd6b15d9f1d8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:15.596652 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d5c293d3ac4392a21282a5a230fdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:15.596616 1087 tablet_service.cc:1467] Processing CreateTablet for tablet b2d5c293d3ac4392a21282a5a230fdc4 (DEFAULT_TABLE table=test-table [id=6ec3d9c12d8b4b0c89dd6b15d9f1d8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:15.597371 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d5c293d3ac4392a21282a5a230fdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:15.599205 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.591718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43831 (local address 127.0.0.1:36264)
0927 02:47:15.592300 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.592308 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.592357 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.592520 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.592521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.592522 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:15.592550 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.592551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.593781 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.593782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.595060 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.595062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.598957 (+ 3895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.598958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.599024 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.599029 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.599095 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.599137 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":53,"thread_start_us":47,"threads_started":1}
I20260927 02:47:15.599241 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.591961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36264 (local address 127.0.98.2:43831)
0927 02:47:15.592118 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.592122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.592363 (+ 241us) server_negotiation.cc:407] Connection header received
0927 02:47:15.592429 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.592431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.592446 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.592516 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:15.592657 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.593686 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.595129 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.595887 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.598943 (+ 3056us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.599015 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.599112 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.599165 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260927 02:47:15.599902 1435 tablet_bootstrap.cc:492] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8: Bootstrap starting.
I20260927 02:47:15.599996 1436 tablet_bootstrap.cc:492] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763: Bootstrap starting.
I20260927 02:47:15.600116 1221 tablet_service.cc:1467] Processing CreateTablet for tablet b2d5c293d3ac4392a21282a5a230fdc4 (DEFAULT_TABLE table=test-table [id=6ec3d9c12d8b4b0c89dd6b15d9f1d8be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:15.600399 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2d5c293d3ac4392a21282a5a230fdc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:15.600546 1436 tablet_bootstrap.cc:654] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:15.600548 1435 tablet_bootstrap.cc:654] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:15.601114 1436 log.cc:824] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:15.601117 1435 log.cc:824] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:15.601751 1435 tablet_bootstrap.cc:492] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8: No bootstrap required, opened a new log
I20260927 02:47:15.601801 1435 ts_tablet_manager.cc:1397] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:15.601845 1436 tablet_bootstrap.cc:492] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763: No bootstrap required, opened a new log
I20260927 02:47:15.601904 1436 ts_tablet_manager.cc:1397] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:15.603166 1435 raft_consensus.cc:348] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.603341 1435 raft_consensus.cc:374] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:15.603382 1435 raft_consensus.cc:729] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a2d44624d94150b946e36dcee792a8, State: Initialized, Role: FOLLOWER
I20260927 02:47:15.603488 1435 consensus_queue.cc:260] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.603559 1439 tablet_bootstrap.cc:492] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54: Bootstrap starting.
I20260927 02:47:15.603756 1435 ts_tablet_manager.cc:1428] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:15.603823 1421 heartbeater.cc:503] Master 127.0.98.62:41107 was elected leader, sending a full tablet report...
I20260927 02:47:15.603801 1436 raft_consensus.cc:348] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.603972 1436 raft_consensus.cc:374] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:15.604012 1436 raft_consensus.cc:729] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e701bdce88254e86a521daaaa2d9a763, State: Initialized, Role: FOLLOWER
I20260927 02:47:15.604116 1436 consensus_queue.cc:260] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.604254 1439 tablet_bootstrap.cc:654] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:15.604386 1436 ts_tablet_manager.cc:1428] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:15.604429 1153 heartbeater.cc:503] Master 127.0.98.62:41107 was elected leader, sending a full tablet report...
I20260927 02:47:15.604768 1439 log.cc:824] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:15.605918 1439 tablet_bootstrap.cc:492] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54: No bootstrap required, opened a new log
I20260927 02:47:15.605976 1439 ts_tablet_manager.cc:1397] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:15.607297 1439 raft_consensus.cc:348] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.607476 1439 raft_consensus.cc:374] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:15.607518 1439 raft_consensus.cc:729] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6824e2caf74ab88b1a3f66b51bcf54, State: Initialized, Role: FOLLOWER
I20260927 02:47:15.607635 1439 consensus_queue.cc:260] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.607872 1439 ts_tablet_manager.cc:1428] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:15.607894 1287 heartbeater.cc:503] Master 127.0.98.62:41107 was elected leader, sending a full tablet report...
I20260927 02:47:15.638232 1440 raft_consensus.cc:482] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:15.638356 1440 raft_consensus.cc:504] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.638753 1440 leader_election.cc:288] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e701bdce88254e86a521daaaa2d9a763 (127.0.98.1:36153), 3b6824e2caf74ab88b1a3f66b51bcf54 (127.0.98.2:43831)
I20260927 02:47:15.641379 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.639016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53393 (local address 127.0.98.1:36153)
0927 02:47:15.639058 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.639060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.639069 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:15.639121 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.639122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.639135 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.639182 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:15.639319 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.639790 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.640708 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.641072 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.641131 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.641142 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.641276 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.641314 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:47:15.641391 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.638939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36153 (local address 127.0.98.3:53393)
0927 02:47:15.638993 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.639005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.639059 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.639190 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.639191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.639192 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:15.639259 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.639261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.639862 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.639863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.640606 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.640608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.641145 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.641146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.641213 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.641216 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.641260 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.641301 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:47:15.641407 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.639052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55607 (local address 127.0.98.2:43831)
0927 02:47:15.639094 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.639095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.639188 (+ 93us) server_negotiation.cc:407] Connection header received
0927 02:47:15.639243 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.639244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.639268 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.639306 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:15.639430 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.639876 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.640746 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.641109 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.641165 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.641175 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.641303 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.641341 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:47:15.641471 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.638939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43831 (local address 127.0.98.3:55607)
0927 02:47:15.639127 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:15.639135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:15.639186 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:15.639316 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:15.639316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:15.639318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:15.639377 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.639379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.639943 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.639944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.640670 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:15.640671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.641176 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.641177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:15.641242 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.641245 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:15.641292 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:47:15.641330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":52,"thread_start_us":82,"threads_started":1}
I20260927 02:47:15.641716 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d5c293d3ac4392a21282a5a230fdc4" candidate_uuid: "37a2d44624d94150b946e36dcee792a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e701bdce88254e86a521daaaa2d9a763" is_pre_election: true
I20260927 02:47:15.641722 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d5c293d3ac4392a21282a5a230fdc4" candidate_uuid: "37a2d44624d94150b946e36dcee792a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" is_pre_election: true
I20260927 02:47:15.641865 1107 raft_consensus.cc:2454] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a2d44624d94150b946e36dcee792a8 in term 0.
I20260927 02:47:15.641868 1241 raft_consensus.cc:2454] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37a2d44624d94150b946e36dcee792a8 in term 0.
I20260927 02:47:15.642098 1308 leader_election.cc:302] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a2d44624d94150b946e36dcee792a8, e701bdce88254e86a521daaaa2d9a763; no voters:
I20260927 02:47:15.642215 1440 raft_consensus.cc:2781] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:15.642263 1440 raft_consensus.cc:482] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:15.642283 1440 raft_consensus.cc:3037] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:15.642975 1440 raft_consensus.cc:504] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.643102 1440 leader_election.cc:288] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [CANDIDATE]: Term 1 election: Requested vote from peers e701bdce88254e86a521daaaa2d9a763 (127.0.98.1:36153), 3b6824e2caf74ab88b1a3f66b51bcf54 (127.0.98.2:43831)
I20260927 02:47:15.643280 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d5c293d3ac4392a21282a5a230fdc4" candidate_uuid: "37a2d44624d94150b946e36dcee792a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e701bdce88254e86a521daaaa2d9a763"
I20260927 02:47:15.643278 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2d5c293d3ac4392a21282a5a230fdc4" candidate_uuid: "37a2d44624d94150b946e36dcee792a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54"
I20260927 02:47:15.643366 1107 raft_consensus.cc:3037] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:15.643368 1241 raft_consensus.cc:3037] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:15.644238 1241 raft_consensus.cc:2454] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a2d44624d94150b946e36dcee792a8 in term 1.
I20260927 02:47:15.644238 1107 raft_consensus.cc:2454] T b2d5c293d3ac4392a21282a5a230fdc4 P e701bdce88254e86a521daaaa2d9a763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a2d44624d94150b946e36dcee792a8 in term 1.
I20260927 02:47:15.644403 1311 leader_election.cc:302] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a2d44624d94150b946e36dcee792a8, 3b6824e2caf74ab88b1a3f66b51bcf54; no voters:
I20260927 02:47:15.644482 1440 raft_consensus.cc:2781] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:15.644624 1440 raft_consensus.cc:686] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [term 1 LEADER]: Becoming Leader. State: Replica: 37a2d44624d94150b946e36dcee792a8, State: Running, Role: LEADER
I20260927 02:47:15.644709 1440 consensus_queue.cc:237] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } }
I20260927 02:47:15.645535 961 catalog_manager.cc:5495] T b2d5c293d3ac4392a21282a5a230fdc4 P 37a2d44624d94150b946e36dcee792a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37a2d44624d94150b946e36dcee792a8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "37a2d44624d94150b946e36dcee792a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e701bdce88254e86a521daaaa2d9a763" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37a2d44624d94150b946e36dcee792a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b6824e2caf74ab88b1a3f66b51bcf54" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43831 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:15.656682 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.649776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.0.98.1:36153)
0927 02:47:15.649820 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.649822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.649955 (+ 133us) server_negotiation.cc:407] Connection header received
0927 02:47:15.649974 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.649975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.649985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.650020 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.650521 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.651031 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.651697 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.651878 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.651930 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.654016 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.654025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.654025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.654031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.655493 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.655863 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.655866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.655866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.655918 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.656083 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.656084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.656084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.656261 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.656390 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.656449 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.656523 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 02:47:15.668520 1288 tablet.cc:2369] T b2d5c293d3ac4392a21282a5a230fdc4 P 3b6824e2caf74ab88b1a3f66b51bcf54: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:15.677488 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.671178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.0.98.1:36153)
0927 02:47:15.671238 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.671241 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.671331 (+ 90us) server_negotiation.cc:407] Connection header received
0927 02:47:15.671381 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.671382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.671391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.671440 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.671963 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.672427 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.673122 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.673311 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.673365 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.675415 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.675416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.675416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.675423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.676553 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.676875 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.676876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.676877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.676928 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.677083 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.677084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.677084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.677170 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.677235 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.677363 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.677422 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:15.698658 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:15.692225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.0.98.1:36153)
0927 02:47:15.692286 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:15.692288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:15.692368 (+ 80us) server_negotiation.cc:407] Connection header received
0927 02:47:15.692422 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:15.692422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:15.692431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:15.692477 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:15.692965 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.693426 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.694527 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:15.694718 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:15.694771 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:15.696852 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:15.696853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:15.696854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:15.696861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:15.697899 (+ 1038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.698173 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.698174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.698174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.698216 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:15.698332 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:15.698332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:15.698332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:15.698411 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:15.698472 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:15.698543 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:47:15.698594 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:47:15.699086 1067 tablet_service.cc:2908] Rejecting scan request for tablet b2d5c293d3ac4392a21282a5a230fdc4: Uninitialized: clean time has not yet been initialized
I20260927 02:47:15.700091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 1024
I20260927 02:47:15.704564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 1158
I20260927 02:47:15.708330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 1292
I20260927 02:47:15.712168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 930
2026-09-27T02:47:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1838 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[1459](info): set up 2 sockets
Sep 27 02:47:15 dist-test-slave-f4rg krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260927 02:47:16.715063 392 mini_kdc.cc:121] Time spent starting KDC: real 0.983s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 02:47:16.878328 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.163s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 02:47:17.085759 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.207s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 02:47:17.313149 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.227s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/test-user.keytab.
W20260927 02:47:18.115556 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.802s user 0.000s sys 0.001s
Sep 27 02:47:18 dist-test-slave-f4rg krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260927 02:47:18.854096 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.720s user 0.000s sys 0.002s
I20260927 02:47:18.854480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32789
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:18.937500 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:18.937709 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:18.937731 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:18.938911 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:18.938952 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:47:18.938970 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:18.938995 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:18.939018 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:18.939039 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:18.940589 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32789
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:18.940814 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:18.941025 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:18.945570 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:18.945663 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:18.945791 1472 server_base.cc:1034] running on GCE node
W20260927 02:47:18.945605 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:18.946069 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:18.946358 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:18.947495 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790477238947473 us; error 35 us; skew 500 ppm
Sep 27 02:47:18 dist-test-slave-f4rg krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:18.953140 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:18.953686 1472 webserver.cc:460] Webserver started at http://127.0.98.62:39389/ using document root <none> and password file <none>
I20260927 02:47:18.953934 1472 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:18.953989 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:18.954121 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:18.955264 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "c4fc4e00c2d94e6b8b9a916b52e26981"
format_stamp: "Formatted at 2026-09-27 02:47:18 on dist-test-slave-f4rg"
server_key: "6fb01d8f607510418d40896f7564f708"
server_key_iv: "f0e052276d4e0aa8d8a75be759daff61"
server_key_version: "encryptionkey@0"
I20260927 02:47:18.955593 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4fc4e00c2d94e6b8b9a916b52e26981"
format_stamp: "Formatted at 2026-09-27 02:47:18 on dist-test-slave-f4rg"
server_key: "6fb01d8f607510418d40896f7564f708"
server_key_iv: "f0e052276d4e0aa8d8a75be759daff61"
server_key_version: "encryptionkey@0"
I20260927 02:47:18.956940 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:18.957640 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:18.957787 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:18.957870 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "c4fc4e00c2d94e6b8b9a916b52e26981"
format_stamp: "Formatted at 2026-09-27 02:47:18 on dist-test-slave-f4rg"
server_key: "6fb01d8f607510418d40896f7564f708"
server_key_iv: "f0e052276d4e0aa8d8a75be759daff61"
server_key_version: "encryptionkey@0"
I20260927 02:47:18.957933 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:18.981527 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:18.983824 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:18.983968 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:18.988494 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32789
I20260927 02:47:18.988543 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32789 every 8 connection(s)
I20260927 02:47:18.988878 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:18.989388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 1472
I20260927 02:47:18.989493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:18.989670 392 external_mini_cluster.cc:1428] Setting key 459a37a54a5f3a6ba76aa3455f4edd22
I20260927 02:47:18.989745 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:18.992533 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981: Bootstrap starting.
I20260927 02:47:18.993134 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:18.993402 1550 log.cc:824] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:18.994083 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981: No bootstrap required, opened a new log
Sep 27 02:47:18 dist-test-slave-f4rg krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:18.995918 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:47:18.996081 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:18.996125 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fc4e00c2d94e6b8b9a916b52e26981, State: Initialized, Role: FOLLOWER
I20260927 02:47:18.996249 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:47:18.996311 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:18.996341 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:18.996387 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:18.999285 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:18.990194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.0.98.62:32789)
0927 02:47:18.990378 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:18.990387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:18.990412 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:47:18.990870 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:18.990878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:18.990929 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:18.991058 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:18.991687 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:18.992624 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:18.993424 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:18.994054 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:18.994163 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:18.996353 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:18.996358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:18.996358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:18.996370 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:18.997869 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:18.998201 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:18.998204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:18.998204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:18.998288 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:18.998452 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:18.998453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:18.998454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:18.998694 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:18.998848 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:18.998970 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:47:18.999111 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260927 02:47:19.033468 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:47:19.033632 1550 leader_election.cc:302] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4fc4e00c2d94e6b8b9a916b52e26981; no voters:
I20260927 02:47:19.033872 1550 leader_election.cc:288] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:19.033948 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:19.034173 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:19.034112 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [term 1 LEADER]: Becoming Leader. State: Replica: c4fc4e00c2d94e6b8b9a916b52e26981, State: Running, Role: LEADER
I20260927 02:47:19.034521 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } }
I20260927 02:47:19.034952 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4fc4e00c2d94e6b8b9a916b52e26981. Latest consensus state: current_term: 1 leader_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } } }
I20260927 02:47:19.035038 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:19.035228 1564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:19.035459 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fc4e00c2d94e6b8b9a916b52e26981" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32789 } } }
I20260927 02:47:19.035537 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:19.035754 1564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:19.038765 1564 catalog_manager.cc:1347] Generated new cluster ID: 894828af14ad41939ac0e5eb455acc02
I20260927 02:47:19.038836 1564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:19.048755 1564 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:19.049490 1564 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:19.055970 1564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981: Generated new TSK 0
I20260927 02:47:19.056171 1564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260927 02:47:19.636647 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.572s user 0.000s sys 0.003s
I20260927 02:47:19.636942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:19.717875 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:19.718088 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:19.718122 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:19.719326 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:19.719381 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:19.719409 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:47:19.719432 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:19.721028 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:19.721289 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:19.721513 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:19.726402 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:19.726413 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:19.726528 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:19.726681 1576 server_base.cc:1034] running on GCE node
I20260927 02:47:19.726981 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:19.727188 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:19.728332 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790477239728310 us; error 35 us; skew 500 ppm
Sep 27 02:47:19 dist-test-slave-f4rg krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:19.730935 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:19.731434 1576 webserver.cc:460] Webserver started at http://127.0.98.1:45447/ using document root <none> and password file <none>
I20260927 02:47:19.731645 1576 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:19.731693 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:19.731822 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:19.732877 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "009831c346cc4bd8ab173a710f601a14"
format_stamp: "Formatted at 2026-09-27 02:47:19 on dist-test-slave-f4rg"
server_key: "c002907235abfa38067b5444804c42c1"
server_key_iv: "23ff739eda6c11109a632ec32002e96d"
server_key_version: "encryptionkey@0"
I20260927 02:47:19.733219 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "009831c346cc4bd8ab173a710f601a14"
format_stamp: "Formatted at 2026-09-27 02:47:19 on dist-test-slave-f4rg"
server_key: "c002907235abfa38067b5444804c42c1"
server_key_iv: "23ff739eda6c11109a632ec32002e96d"
server_key_version: "encryptionkey@0"
I20260927 02:47:19.734473 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:47:19.735239 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:19.735388 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:19.735474 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "009831c346cc4bd8ab173a710f601a14"
format_stamp: "Formatted at 2026-09-27 02:47:19 on dist-test-slave-f4rg"
server_key: "c002907235abfa38067b5444804c42c1"
server_key_iv: "23ff739eda6c11109a632ec32002e96d"
server_key_version: "encryptionkey@0"
I20260927 02:47:19.735540 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:19.743353 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:19.745075 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:19.745201 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:19.745661 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:19.745703 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:19.745730 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:19.745764 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:19.751760 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35441
I20260927 02:47:19.751878 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35441 every 8 connection(s)
I20260927 02:47:19.752143 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:47:19 dist-test-slave-f4rg krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:19.755484 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:19.746083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46481 (local address 127.0.98.62:32789)
0927 02:47:19.746215 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:19.746217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:19.746323 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:47:19.746853 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:19.746854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:19.746868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:19.746953 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:19.747809 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.748577 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.750047 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.750267 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.750325 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:19.752925 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:19.752926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:19.752926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:19.752933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:19.754117 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:19.754490 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:19.754492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:19.754492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:19.754604 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:19.754852 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:19.754853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:19.754853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:19.754974 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:19.755030 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:19.755293 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 02:47:19.755389 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260927 02:47:19.755635 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:19.746052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.1:46481)
0927 02:47:19.746197 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:19.746238 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:19.746750 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:19.747024 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:19.747028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:19.747422 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:19.747707 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:19.747713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.748686 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.748687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:19.749971 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:19.749974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.750345 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.750346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:19.750483 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:19.751100 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:19.751110 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:19.752784 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:19.754192 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:19.754198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:19.754200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:19.754429 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:19.754676 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:19.754677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:19.754677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:19.754775 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:19.755028 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:19.755035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:19.755167 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 02:47:19.755263 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:47:19.755435 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260927 02:47:19.756091 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32789
I20260927 02:47:19.756176 1708 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:19.756367 1708 heartbeater.cc:511] Master 127.0.98.62:32789 requested a full tablet report, sending...
I20260927 02:47:19.756901 1503 ts_manager.cc:194] Registered new tserver with Master: 009831c346cc4bd8ab173a710f601a14 (127.0.98.1:35441)
I20260927 02:47:19.757565 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46481
I20260927 02:47:19.761307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 1576
I20260927 02:47:19.761390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:19.761535 392 external_mini_cluster.cc:1428] Setting key ea28ba581f81d0122c517e6eaa6668eb
I20260927 02:47:19.762048 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:19.757193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40293 (local address 127.0.98.62:32789)
0927 02:47:19.757230 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:19.757232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:19.757347 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:47:19.757395 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:19.757396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:19.757406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:19.757448 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:19.757754 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.758282 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.759035 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.759275 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.759338 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:19.760089 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:19.760090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:19.760090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:19.760097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:19.761051 (+ 954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:19.761385 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:19.761386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:19.761386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:19.761440 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:19.761608 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:19.761609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:19.761610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:19.761709 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:19.761762 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:19.761893 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:47:19.761961 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:47:19.762058 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:19.757151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.1:40293)
0927 02:47:19.757284 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:19.757292 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:19.757344 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:19.757451 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:19.757452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:19.757617 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:19.757666 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:19.757668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.758380 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.758381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:19.758969 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:19.758971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:19.759331 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:19.759331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:19.759388 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:19.759736 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:19.759741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:19.760013 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:19.761123 (+ 1110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:19.761124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:19.761124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:19.761298 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:19.761503 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:19.761504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:19.761504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:19.761553 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:19.761762 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:19.761763 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:19.761804 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:47:19.761863 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:47:19.761956 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 02:47:20.421172 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.659s user 0.000s sys 0.003s
I20260927 02:47:20.421557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:20.503353 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:20.503561 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:20.503583 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:20.504752 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:20.504801 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:20.504820 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:47:20.504843 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:20.506482 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:20.506714 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:20.506954 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:20.511832 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:20.512126 1717 server_base.cc:1034] running on GCE node
W20260927 02:47:20.511811 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:20.511843 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:20.512461 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:20.512715 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:20.513893 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790477240513877 us; error 31 us; skew 500 ppm
Sep 27 02:47:20 dist-test-slave-f4rg krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:20.516376 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:20.516819 1717 webserver.cc:460] Webserver started at http://127.0.98.2:37979/ using document root <none> and password file <none>
I20260927 02:47:20.517030 1717 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:20.517073 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:20.517203 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:20.518245 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e44fff5969f456c802955b53039222f"
format_stamp: "Formatted at 2026-09-27 02:47:20 on dist-test-slave-f4rg"
server_key: "8cfad9e03780968353c2878eb4483000"
server_key_iv: "133ead3dc19e8707cbd1336d030d9cb6"
server_key_version: "encryptionkey@0"
I20260927 02:47:20.518571 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e44fff5969f456c802955b53039222f"
format_stamp: "Formatted at 2026-09-27 02:47:20 on dist-test-slave-f4rg"
server_key: "8cfad9e03780968353c2878eb4483000"
server_key_iv: "133ead3dc19e8707cbd1336d030d9cb6"
server_key_version: "encryptionkey@0"
I20260927 02:47:20.519980 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:20.520678 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.520853 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:20.520941 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "1e44fff5969f456c802955b53039222f"
format_stamp: "Formatted at 2026-09-27 02:47:20 on dist-test-slave-f4rg"
server_key: "8cfad9e03780968353c2878eb4483000"
server_key_iv: "133ead3dc19e8707cbd1336d030d9cb6"
server_key_version: "encryptionkey@0"
I20260927 02:47:20.520994 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:20.532641 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:20.534863 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:20.534982 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:20.535480 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:20.535521 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.535542 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:20.535554 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:20.541684 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43545
I20260927 02:47:20.541909 1848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43545 every 8 connection(s)
I20260927 02:47:20.542119 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:47:20 dist-test-slave-f4rg krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:20.545985 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.535965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37595 (local address 127.0.98.62:32789)
0927 02:47:20.536085 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.536087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.536300 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:47:20.536844 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.536845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.536861 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.536950 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.537809 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.538602 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.540129 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.540452 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.540523 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.543324 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.543325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.543325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.543332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.544636 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.545084 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.545084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.545085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.545132 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.545364 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.545365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.545366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.545500 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.545564 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.545838 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.545916 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260927 02:47:20.546120 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.535944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.2:37595)
0927 02:47:20.536170 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.536210 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.536733 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.537040 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.537045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.537388 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:20.537708 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.537716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.538719 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.538719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.540017 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.540020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.540549 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.540550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.541033 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.541578 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:20.541588 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:20.543220 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:20.544750 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.544757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.544758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.545017 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.545176 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.545176 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.545176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.545284 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.545558 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:20.545561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:20.545686 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.545816 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.545940 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260927 02:47:20.546530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 1717
I20260927 02:47:20.546626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:20.546700 1849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32789
I20260927 02:47:20.546793 1849 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:20.546857 392 external_mini_cluster.cc:1428] Setting key a6d0f3ca1daabca979e8ada49e621a2a
I20260927 02:47:20.547003 1849 heartbeater.cc:511] Master 127.0.98.62:32789 requested a full tablet report, sending...
I20260927 02:47:20.547560 1504 ts_manager.cc:194] Registered new tserver with Master: 1e44fff5969f456c802955b53039222f (127.0.98.2:43545)
I20260927 02:47:20.548025 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37595
I20260927 02:47:20.554639 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.548981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.2:44565)
0927 02:47:20.549117 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:20.549126 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:20.549178 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:20.549294 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:20.549294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:20.549462 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:20.549512 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.549513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.550374 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.550375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.550989 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:20.550991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.551423 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.551424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:20.551470 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.551901 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:20.551906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:20.552190 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:20.553708 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.553709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.553709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.553868 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.554055 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:20.554056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:20.554056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:20.554108 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:20.554416 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:20.554416 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:20.554450 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:47:20.554490 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:47:20.554564 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260927 02:47:20.554697 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:20.549030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44565 (local address 127.0.98.62:32789)
0927 02:47:20.549067 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:20.549069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:20.549178 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:47:20.549229 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:20.549230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:20.549239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:20.549287 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:20.549575 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.550270 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.551055 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:20.551355 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:20.551427 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:20.552267 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:20.552269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:20.552269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:20.552276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:20.553634 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.553935 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.553936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.553937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.553995 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:20.554159 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:20.554160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:20.554161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:20.554355 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:20.554429 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:20.554529 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:47:20.554614 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 02:47:20.758400 1708 heartbeater.cc:503] Master 127.0.98.62:32789 was elected leader, sending a full tablet report...
I20260927 02:47:21.549513 1849 heartbeater.cc:503] Master 127.0.98.62:32789 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 02:47:21.590950 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.043s user 0.001s sys 0.001s
I20260927 02:47:21.591307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32789
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:21.669523 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:21.669759 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:21.669782 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:21.670984 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:21.671030 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:21.671051 1858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:47:21.671067 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:21.672590 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:21.672812 1858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:21.673025 1858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:21.677793 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:21.677766 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:21.677948 1858 server_base.cc:1034] running on GCE node
W20260927 02:47:21.677783 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:21.678207 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:21.678437 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:21.679575 1858 hybrid_clock.cc:648] HybridClock initialized: now 1790477241679565 us; error 26 us; skew 500 ppm
Sep 27 02:47:21 dist-test-slave-f4rg krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:21.682159 1858 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:21.682585 1858 webserver.cc:460] Webserver started at http://127.0.98.3:34721/ using document root <none> and password file <none>
I20260927 02:47:21.682785 1858 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:21.682857 1858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:21.682977 1858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:21.684064 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "6996750496d941f8bc1743b398fc22bb"
format_stamp: "Formatted at 2026-09-27 02:47:21 on dist-test-slave-f4rg"
server_key: "e869a0b4915f7c963a3b62c08f030f7c"
server_key_iv: "c902782857f668b8b37bf20d06814adf"
server_key_version: "encryptionkey@0"
I20260927 02:47:21.684387 1858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6996750496d941f8bc1743b398fc22bb"
format_stamp: "Formatted at 2026-09-27 02:47:21 on dist-test-slave-f4rg"
server_key: "e869a0b4915f7c963a3b62c08f030f7c"
server_key_iv: "c902782857f668b8b37bf20d06814adf"
server_key_version: "encryptionkey@0"
I20260927 02:47:21.685613 1858 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:21.686275 1873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:21.686400 1858 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:21.686478 1858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "6996750496d941f8bc1743b398fc22bb"
format_stamp: "Formatted at 2026-09-27 02:47:21 on dist-test-slave-f4rg"
server_key: "e869a0b4915f7c963a3b62c08f030f7c"
server_key_iv: "c902782857f668b8b37bf20d06814adf"
server_key_version: "encryptionkey@0"
I20260927 02:47:21.686537 1858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:21.693162 1858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:21.695344 1858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:21.695474 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:21.695925 1858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:21.695967 1858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:21.695994 1858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:21.696012 1858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:21.702351 1858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38791
I20260927 02:47:21.702447 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38791 every 8 connection(s)
Sep 27 02:47:21 dist-test-slave-f4rg krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:21.702755 1858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:21.705562 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.696375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40651 (local address 127.0.98.62:32789)
0927 02:47:21.696468 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.696470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.696623 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:47:21.697144 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.697146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.697161 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.697227 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:21.698074 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.698856 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.700451 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.700693 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.700750 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.703200 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:21.703201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:21.703201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:21.703209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:21.704329 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.704682 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.704683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.704683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.704745 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.704964 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.704965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.704965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.705078 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:21.705133 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.705385 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.705467 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":43,"threads_started":1}
I20260927 02:47:21.705600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 1858
I20260927 02:47:21.705690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:21.705714 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.696369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.3:40651)
0927 02:47:21.696509 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.696548 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.697045 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.697335 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.697340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.697676 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:21.697963 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.697970 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.698965 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.698966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.700360 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.700363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.700766 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.700767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.700916 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.701437 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:21.701450 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:21.703113 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:21.704403 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:21.704409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:21.704411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:21.704617 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:21.704797 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:21.704797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:21.704798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:21.704910 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:21.705130 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:21.705136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:21.705264 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.705347 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.705515 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260927 02:47:21.705895 392 external_mini_cluster.cc:1428] Setting key c2438a9ebb7556bc101148eaa5292556
I20260927 02:47:21.706421 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32789
I20260927 02:47:21.706504 1990 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:21.706678 1990 heartbeater.cc:511] Master 127.0.98.62:32789 requested a full tablet report, sending...
I20260927 02:47:21.707011 1503 ts_manager.cc:194] Registered new tserver with Master: 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791)
I20260927 02:47:21.707527 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40651
I20260927 02:47:21.712064 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.707404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37867 (local address 127.0.98.62:32789)
0927 02:47:21.707439 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.707440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.707568 (+ 128us) server_negotiation.cc:407] Connection header received
0927 02:47:21.707618 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.707619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.707628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.707670 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:21.707954 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.708418 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.709067 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.709268 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.709319 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.710076 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:21.710077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:21.710077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:21.710083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:21.711149 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.711424 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.711425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.711425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.711473 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.711625 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.711626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.711626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.711723 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:21.711776 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.711902 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.711974 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:47:21.712077 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.707394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.98.3:37867)
0927 02:47:21.707506 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.707516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.707568 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.707673 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.707673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.707831 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:21.707892 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.707893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.708502 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.708503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.709001 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.709002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.709323 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.709324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.709382 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.709735 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:21.709738 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:21.710002 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:21.711216 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:21.711217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:21.711217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:21.711365 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:21.711523 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:21.711524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:21.711524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:21.711577 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:21.711774 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:21.711775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:21.711814 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.711880 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.711974 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260927 02:47:21.717355 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:47:21 dist-test-slave-f4rg krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:47:21 dist-test-slave-f4rg krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:21.738080 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.731643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.0.98.62:32789)
0927 02:47:21.731704 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.731706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.731717 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:47:21.731790 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.731790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.731799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.731838 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:21.732383 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.732858 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.733548 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.733752 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.733805 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.735992 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:21.735994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:21.735994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:21.736003 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:21.737104 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.737409 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.737410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.737410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.737457 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.737612 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.737613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.737613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.737715 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:21.737800 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.737913 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.737988 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:47:21.739574 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:21.741101 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:21.747135 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.743286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35441 (local address 127.0.0.1:48860)
0927 02:47:21.743589 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.743629 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.743704 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.743875 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.743877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.743879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.743940 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.743942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.744926 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.744927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.746082 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.746085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.746852 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.746853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.746923 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.746936 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.747014 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.747063 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":103,"threads_started":1}
I20260927 02:47:21.747171 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.743405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.0.98.1:35441)
0927 02:47:21.743531 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.743536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.743686 (+ 150us) server_negotiation.cc:407] Connection header received
0927 02:47:21.743776 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.743786 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.743802 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.743869 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.744010 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.744837 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.746169 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.746757 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.746847 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.746910 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.747064 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.747104 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":43,"threads_started":1}
I20260927 02:47:21.747452 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.743404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.0.98.3:38791)
0927 02:47:21.743638 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.743642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.743782 (+ 140us) server_negotiation.cc:407] Connection header received
0927 02:47:21.743824 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.743828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.743843 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.743907 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.744091 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.745191 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.746190 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.746781 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.746899 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.746973 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.747314 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.747350 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":77,"threads_started":1}
I20260927 02:47:21.747505 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.743532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43545 (local address 127.0.0.1:33974)
0927 02:47:21.743807 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.743816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.743859 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.744102 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.744103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.744105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.744140 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.744141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.745325 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.745326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.746402 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.746404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.747307 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.747308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.747361 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.747364 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.747417 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.747449 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":56,"threads_started":1}
I20260927 02:47:21.747570 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.743627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.98.2:43545)
0927 02:47:21.743763 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.743767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.743934 (+ 167us) server_negotiation.cc:407] Connection header received
0927 02:47:21.743966 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.743968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.743982 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.744045 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.744204 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.745250 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.746479 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.747175 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.747229 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.747276 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.747452 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.747493 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":52,"threads_started":1}
I20260927 02:47:21.747252 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.743302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38791 (local address 127.0.0.1:49414)
0927 02:47:21.743634 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.743644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.743704 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.743969 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.743970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.743972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.744025 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.744027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.745277 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.745278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.746115 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.746117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.746998 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.746999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.747094 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.747098 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.747146 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.747183 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":90,"threads_started":1}
I20260927 02:47:21.748206 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 5511bd19f55741488ea8004b1633e689 (DEFAULT_TABLE table=test-table [id=abb7303bfee941b58e5a1d7d25cc4214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:21.748580 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5511bd19f55741488ea8004b1633e689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.748538 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 5511bd19f55741488ea8004b1633e689 (DEFAULT_TABLE table=test-table [id=abb7303bfee941b58e5a1d7d25cc4214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:21.748869 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5511bd19f55741488ea8004b1633e689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.751049 2014 tablet_bootstrap.cc:492] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Bootstrap starting.
I20260927 02:47:21.751367 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 5511bd19f55741488ea8004b1633e689 (DEFAULT_TABLE table=test-table [id=abb7303bfee941b58e5a1d7d25cc4214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:21.751681 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5511bd19f55741488ea8004b1633e689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.751693 2015 tablet_bootstrap.cc:492] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Bootstrap starting.
I20260927 02:47:21.751718 2014 tablet_bootstrap.cc:654] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.752054 2014 log.cc:824] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:21.752259 2015 tablet_bootstrap.cc:654] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.752544 2015 log.cc:824] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:21.752707 2014 tablet_bootstrap.cc:492] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: No bootstrap required, opened a new log
I20260927 02:47:21.752763 2014 ts_tablet_manager.cc:1397] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:21.753206 2015 tablet_bootstrap.cc:492] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: No bootstrap required, opened a new log
I20260927 02:47:21.753265 2015 ts_tablet_manager.cc:1397] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:21.754101 2014 raft_consensus.cc:348] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.754218 2014 raft_consensus.cc:374] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.754237 2014 raft_consensus.cc:729] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009831c346cc4bd8ab173a710f601a14, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.754334 2014 consensus_queue.cc:260] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.754547 2014 ts_tablet_manager.cc:1428] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:21.754710 2019 tablet_bootstrap.cc:492] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Bootstrap starting.
I20260927 02:47:21.755187 2015 raft_consensus.cc:348] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.755306 2019 tablet_bootstrap.cc:654] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.755347 2015 raft_consensus.cc:374] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.755400 2015 raft_consensus.cc:729] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6996750496d941f8bc1743b398fc22bb, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.755505 2015 consensus_queue.cc:260] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.755574 2019 log.cc:824] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:21.755741 2015 ts_tablet_manager.cc:1428] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:47:21.755788 1990 heartbeater.cc:503] Master 127.0.98.62:32789 was elected leader, sending a full tablet report...
I20260927 02:47:21.756278 2019 tablet_bootstrap.cc:492] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: No bootstrap required, opened a new log
I20260927 02:47:21.756336 2019 ts_tablet_manager.cc:1397] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:21.758117 2019 raft_consensus.cc:348] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.758234 2019 raft_consensus.cc:374] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.758265 2019 raft_consensus.cc:729] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e44fff5969f456c802955b53039222f, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.758364 2019 consensus_queue.cc:260] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.758548 2019 ts_tablet_manager.cc:1428] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:21.764694 2018 raft_consensus.cc:482] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:21.764772 2018 raft_consensus.cc:504] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.765064 2018 leader_election.cc:288] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791), 1e44fff5969f456c802955b53039222f (127.0.98.2:43545)
I20260927 02:47:21.767802 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.765251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55891 (local address 127.0.98.2:43545)
0927 02:47:21.765292 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.765294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.765462 (+ 168us) server_negotiation.cc:407] Connection header received
0927 02:47:21.765516 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.765517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.765528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.765571 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.765729 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.766190 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.767195 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.767534 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.767591 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.767601 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.767711 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.767743 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:47:21.767832 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.765203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43545 (local address 127.0.98.1:55891)
0927 02:47:21.765404 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.765414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.765470 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.765587 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.765587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.765589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.765662 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.765664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.766257 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.766258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.767122 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.767124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.767597 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.767597 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.767657 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.767660 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.767702 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.767738 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":30,"thread_start_us":102,"threads_started":1}
I20260927 02:47:21.767869 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.765341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59445 (local address 127.0.98.3:38791)
0927 02:47:21.765389 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.765390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.765485 (+ 95us) server_negotiation.cc:407] Connection header received
0927 02:47:21.765579 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.765580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.765589 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.765632 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.765768 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.766215 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.767202 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.767570 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.767627 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.767636 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.767775 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.767809 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:47:21.768034 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.765211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38791 (local address 127.0.98.1:59445)
0927 02:47:21.765416 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.765425 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.765509 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.765646 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.765647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.765649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.765705 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.765707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.766285 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.766286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.767129 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.767132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.767656 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.767657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.767718 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.767721 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.767764 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.767801 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260927 02:47:21.768137 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511bd19f55741488ea8004b1633e689" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e44fff5969f456c802955b53039222f" is_pre_election: true
I20260927 02:47:21.768262 1803 raft_consensus.cc:2454] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 0.
I20260927 02:47:21.768306 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511bd19f55741488ea8004b1633e689" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996750496d941f8bc1743b398fc22bb" is_pre_election: true
I20260927 02:47:21.768435 1944 raft_consensus.cc:2454] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 0.
I20260927 02:47:21.768450 1595 leader_election.cc:302] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009831c346cc4bd8ab173a710f601a14, 1e44fff5969f456c802955b53039222f; no voters:
I20260927 02:47:21.768543 2018 raft_consensus.cc:2781] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:21.768595 2018 raft_consensus.cc:482] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:21.768623 2018 raft_consensus.cc:3037] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.769254 2018 raft_consensus.cc:504] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.769361 2018 leader_election.cc:288] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 election: Requested vote from peers 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791), 1e44fff5969f456c802955b53039222f (127.0.98.2:43545)
I20260927 02:47:21.769490 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511bd19f55741488ea8004b1633e689" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996750496d941f8bc1743b398fc22bb"
I20260927 02:47:21.769505 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5511bd19f55741488ea8004b1633e689" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e44fff5969f456c802955b53039222f"
I20260927 02:47:21.769559 1944 raft_consensus.cc:3037] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.769560 1803 raft_consensus.cc:3037] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.770200 1944 raft_consensus.cc:2454] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 1.
I20260927 02:47:21.770221 1803 raft_consensus.cc:2454] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 1.
I20260927 02:47:21.770358 1594 leader_election.cc:302] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009831c346cc4bd8ab173a710f601a14, 6996750496d941f8bc1743b398fc22bb; no voters:
I20260927 02:47:21.770447 2018 raft_consensus.cc:2781] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:21.770572 2018 raft_consensus.cc:686] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 1 LEADER]: Becoming Leader. State: Replica: 009831c346cc4bd8ab173a710f601a14, State: Running, Role: LEADER
I20260927 02:47:21.770646 2018 consensus_queue.cc:237] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } }
I20260927 02:47:21.771382 1503 catalog_manager.cc:5495] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009831c346cc4bd8ab173a710f601a14 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "009831c346cc4bd8ab173a710f601a14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:21.784255 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.782173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.0.98.1:35441)
0927 02:47:21.782223 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.782225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.782234 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:21.782305 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.782306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.782323 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.782376 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:21.782466 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.782965 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.783663 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.783848 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.783893 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.784025 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:21.784110 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:21.784158 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.784185 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.784200 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:47:21.785786 1803 raft_consensus.cc:1261] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Refusing update from remote peer 009831c346cc4bd8ab173a710f601a14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:21.785792 1944 raft_consensus.cc:1261] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Refusing update from remote peer 009831c346cc4bd8ab173a710f601a14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:21.786087 2025 consensus_queue.cc:1035] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:21.786198 2018 consensus_queue.cc:1035] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:21.788200 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333794782352302080 to 7333794782293098496. Current Snapshot: MvccSnapshot[applied={T|T < 7333794782352302080}]
I20260927 02:47:21.788425 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333794782352302080 to 7333794782293098496. Current Snapshot: MvccSnapshot[applied={T|T < 7333794782352302080}]
I20260927 02:47:21.788584 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333794782352302080 to 7333794782293098496. Current Snapshot: MvccSnapshot[applied={T|T < 7333794782352302080}]
I20260927 02:47:21.792234 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:21.792317 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:21.792999 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981: Sending DeleteTablet for 3 replicas of tablet 5511bd19f55741488ea8004b1633e689
I20260927 02:47:21.793295 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 5511bd19f55741488ea8004b1633e689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49414
I20260927 02:47:21.793336 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 5511bd19f55741488ea8004b1633e689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48860
I20260927 02:47:21.793401 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 5511bd19f55741488ea8004b1633e689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33974
I20260927 02:47:21.793599 2040 tablet_replica.cc:330] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: stopping tablet replica
I20260927 02:47:21.793725 2038 tablet_replica.cc:330] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: stopping tablet replica
I20260927 02:47:21.793727 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:47:21.793826 2038 raft_consensus.cc:2229] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 02:47:21.793871 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:21.794188 2038 raft_consensus.cc:2258] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:21.794343 2040 raft_consensus.cc:2229] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:21.794503 2040 raft_consensus.cc:2258] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:21.795002 2040 ts_tablet_manager.cc:1905] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:21.795084 1924 tablet_service.cc:1467] Processing CreateTablet for tablet bfcfcc3fb26e4506bb460697570bdc75 (DEFAULT_TABLE table=test-table [id=3b3d589ab6104fdb8cd7864be7b557ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:21.795183 1642 tablet_service.cc:1467] Processing CreateTablet for tablet bfcfcc3fb26e4506bb460697570bdc75 (DEFAULT_TABLE table=test-table [id=3b3d589ab6104fdb8cd7864be7b557ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:21.795207 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcfcc3fb26e4506bb460697570bdc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.795270 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcfcc3fb26e4506bb460697570bdc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.795349 2038 ts_tablet_manager.cc:1905] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:21.795034 1783 tablet_service.cc:1467] Processing CreateTablet for tablet bfcfcc3fb26e4506bb460697570bdc75 (DEFAULT_TABLE table=test-table [id=3b3d589ab6104fdb8cd7864be7b557ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:21.796452 2015 tablet_bootstrap.cc:492] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: Bootstrap starting.
I20260927 02:47:21.796452 2014 tablet_bootstrap.cc:492] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: Bootstrap starting.
I20260927 02:47:21.796487 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfcfcc3fb26e4506bb460697570bdc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.796829 2014 tablet_bootstrap.cc:654] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.796830 2015 tablet_bootstrap.cc:654] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.797859 2040 ts_tablet_manager.cc:1918] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:21.797941 2040 log.cc:1196] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/5511bd19f55741488ea8004b1633e689
I20260927 02:47:21.798171 2040 ts_tablet_manager.cc:1939] T 5511bd19f55741488ea8004b1633e689 P 1e44fff5969f456c802955b53039222f: Deleting consensus metadata
I20260927 02:47:21.798343 2038 ts_tablet_manager.cc:1918] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:21.798465 2038 log.cc:1196] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/5511bd19f55741488ea8004b1633e689
I20260927 02:47:21.798741 2038 ts_tablet_manager.cc:1939] T 5511bd19f55741488ea8004b1633e689 P 6996750496d941f8bc1743b398fc22bb: Deleting consensus metadata
I20260927 02:47:21.798892 2015 tablet_bootstrap.cc:492] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: No bootstrap required, opened a new log
I20260927 02:47:21.798944 2015 ts_tablet_manager.cc:1397] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:47:21.798210 2019 tablet_bootstrap.cc:492] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: Bootstrap starting.
I20260927 02:47:21.799079 2015 raft_consensus.cc:348] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.799141 2015 raft_consensus.cc:374] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.799160 2015 raft_consensus.cc:729] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6996750496d941f8bc1743b398fc22bb, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.799154 1490 catalog_manager.cc:4841] TS 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791): tablet 5511bd19f55741488ea8004b1633e689 (table test-table [id=abb7303bfee941b58e5a1d7d25cc4214]) successfully deleted
I20260927 02:47:21.799201 2015 consensus_queue.cc:260] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.799288 2015 ts_tablet_manager.cc:1428] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:21.799422 2039 tablet_replica.cc:330] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: stopping tablet replica
I20260927 02:47:21.799466 2019 tablet_bootstrap.cc:654] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.799804 1491 catalog_manager.cc:4841] TS 1e44fff5969f456c802955b53039222f (127.0.98.2:43545): tablet 5511bd19f55741488ea8004b1633e689 (table test-table [id=abb7303bfee941b58e5a1d7d25cc4214]) successfully deleted
I20260927 02:47:21.799821 2039 raft_consensus.cc:2229] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:21.799978 2039 raft_consensus.cc:2258] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:21.800148 2019 tablet_bootstrap.cc:492] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: No bootstrap required, opened a new log
I20260927 02:47:21.800194 2019 ts_tablet_manager.cc:1397] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:21.800405 2039 ts_tablet_manager.cc:1905] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:21.800424 2014 tablet_bootstrap.cc:492] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: No bootstrap required, opened a new log
I20260927 02:47:21.800437 2019 raft_consensus.cc:348] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.800482 2014 ts_tablet_manager.cc:1397] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 02:47:21.800498 2019 raft_consensus.cc:374] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.800525 2019 raft_consensus.cc:729] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e44fff5969f456c802955b53039222f, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.800558 2019 consensus_queue.cc:260] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.800638 2019 ts_tablet_manager.cc:1428] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:21.800614 2014 raft_consensus.cc:348] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.800671 2014 raft_consensus.cc:374] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.800689 2014 raft_consensus.cc:729] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009831c346cc4bd8ab173a710f601a14, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.800727 2014 consensus_queue.cc:260] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.800824 2014 ts_tablet_manager.cc:1428] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:21.802105 2039 ts_tablet_manager.cc:1918] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:21.802160 2039 log.cc:1196] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/5511bd19f55741488ea8004b1633e689
I20260927 02:47:21.802314 2039 ts_tablet_manager.cc:1939] T 5511bd19f55741488ea8004b1633e689 P 009831c346cc4bd8ab173a710f601a14: Deleting consensus metadata
I20260927 02:47:21.802700 1491 catalog_manager.cc:4841] TS 009831c346cc4bd8ab173a710f601a14 (127.0.98.1:35441): tablet 5511bd19f55741488ea8004b1633e689 (table test-table [id=abb7303bfee941b58e5a1d7d25cc4214]) successfully deleted
I20260927 02:47:21.896998 2033 raft_consensus.cc:482] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:21.897096 2033 raft_consensus.cc:504] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.897261 2033 leader_election.cc:288] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e44fff5969f456c802955b53039222f (127.0.98.2:43545), 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791)
I20260927 02:47:21.897500 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcfcc3fb26e4506bb460697570bdc75" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e44fff5969f456c802955b53039222f" is_pre_election: true
I20260927 02:47:21.897493 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcfcc3fb26e4506bb460697570bdc75" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996750496d941f8bc1743b398fc22bb" is_pre_election: true
I20260927 02:47:21.897589 1803 raft_consensus.cc:2454] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 0.
I20260927 02:47:21.897596 1944 raft_consensus.cc:2454] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 0.
I20260927 02:47:21.897744 1594 leader_election.cc:302] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009831c346cc4bd8ab173a710f601a14, 6996750496d941f8bc1743b398fc22bb; no voters:
I20260927 02:47:21.897841 2033 raft_consensus.cc:2781] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:21.897872 2033 raft_consensus.cc:482] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:21.897891 2033 raft_consensus.cc:3037] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.898538 2033 raft_consensus.cc:504] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.898648 2033 leader_election.cc:288] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 election: Requested vote from peers 1e44fff5969f456c802955b53039222f (127.0.98.2:43545), 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791)
I20260927 02:47:21.898846 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcfcc3fb26e4506bb460697570bdc75" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996750496d941f8bc1743b398fc22bb"
I20260927 02:47:21.898849 1803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfcfcc3fb26e4506bb460697570bdc75" candidate_uuid: "009831c346cc4bd8ab173a710f601a14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e44fff5969f456c802955b53039222f"
I20260927 02:47:21.898933 1803 raft_consensus.cc:3037] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.898934 1944 raft_consensus.cc:3037] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.899529 1803 raft_consensus.cc:2454] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 1.
I20260927 02:47:21.899672 1595 leader_election.cc:302] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009831c346cc4bd8ab173a710f601a14, 1e44fff5969f456c802955b53039222f; no voters:
I20260927 02:47:21.899752 2033 raft_consensus.cc:2781] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:21.899752 1944 raft_consensus.cc:2454] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009831c346cc4bd8ab173a710f601a14 in term 1.
I20260927 02:47:21.899793 2033 raft_consensus.cc:686] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 1 LEADER]: Becoming Leader. State: Replica: 009831c346cc4bd8ab173a710f601a14, State: Running, Role: LEADER
I20260927 02:47:21.899845 2033 consensus_queue.cc:237] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.900439 1503 catalog_manager.cc:5495] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009831c346cc4bd8ab173a710f601a14 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "009831c346cc4bd8ab173a710f601a14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:47:21 dist-test-slave-f4rg krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:21.945596 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.939557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.0.98.62:32789)
0927 02:47:21.939628 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.939630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.939739 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:47:21.939799 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.939799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.939808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.939855 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:21.940310 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.940808 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.941465 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.941636 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.941691 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.943585 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:21.943586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:21.943587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:21.943594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:21.944598 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.944919 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.944920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.944920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.944967 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:21.945114 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:21.945115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:21.945115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:21.945215 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:21.945267 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.945405 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.945484 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:47:21.945647 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.939513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32789 (local address 127.0.0.1:50192)
0927 02:47:21.939661 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.939671 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.939741 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.939865 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.939866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.940182 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:21.940228 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.940231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.940886 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.940886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.941398 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.941399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.941692 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.941693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.941736 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.942189 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:21.942193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:21.943507 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:21.944665 (+ 1158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:21.944673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:21.944675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:21.944852 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:21.945017 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:21.945018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:21.945018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:21.945069 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:21.945264 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:21.945270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:21.945315 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.945372 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.945531 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":65,"thread_start_us":78,"threads_started":1}
I20260927 02:47:21.947438 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50192:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:47:21.947623 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:21.948845 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe912edabad494f9ddee26532234513 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=981bb988de32456797b0e722b13d1a47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:47:21.948844 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe912edabad494f9ddee26532234513 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=981bb988de32456797b0e722b13d1a47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:47:21.948997 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe912edabad494f9ddee26532234513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.948995 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe912edabad494f9ddee26532234513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.950430 2015 tablet_bootstrap.cc:492] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: Bootstrap starting.
I20260927 02:47:21.950438 2014 tablet_bootstrap.cc:492] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: Bootstrap starting.
I20260927 02:47:21.950531 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe912edabad494f9ddee26532234513 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=981bb988de32456797b0e722b13d1a47]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:47:21.950659 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe912edabad494f9ddee26532234513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:21.950795 2014 tablet_bootstrap.cc:654] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.950835 2015 tablet_bootstrap.cc:654] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.951370 2014 tablet_bootstrap.cc:492] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: No bootstrap required, opened a new log
I20260927 02:47:21.951421 2014 ts_tablet_manager.cc:1397] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:47:21.951526 2014 raft_consensus.cc:348] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.951583 2014 raft_consensus.cc:374] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.951601 2014 raft_consensus.cc:729] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009831c346cc4bd8ab173a710f601a14, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.951642 2014 consensus_queue.cc:260] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.951730 2014 ts_tablet_manager.cc:1428] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:21.951799 2019 tablet_bootstrap.cc:492] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: Bootstrap starting.
I20260927 02:47:21.951818 2033 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.951882 2033 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:21.952200 2015 tablet_bootstrap.cc:492] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: No bootstrap required, opened a new log
I20260927 02:47:21.952205 2019 tablet_bootstrap.cc:654] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:21.952240 2015 ts_tablet_manager.cc:1397] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:47:21.952368 2015 raft_consensus.cc:348] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.952425 2015 raft_consensus.cc:374] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.952442 2015 raft_consensus.cc:729] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6996750496d941f8bc1743b398fc22bb, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.952487 2015 consensus_queue.cc:260] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.952567 2015 ts_tablet_manager.cc:1428] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:21.952685 2020 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.952759 2020 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:21.952776 2019 tablet_bootstrap.cc:492] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: No bootstrap required, opened a new log
I20260927 02:47:21.952850 2019 ts_tablet_manager.cc:1397] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:21.952978 2019 raft_consensus.cc:348] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.953050 2019 raft_consensus.cc:374] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:21.953068 2019 raft_consensus.cc:729] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e44fff5969f456c802955b53039222f, State: Initialized, Role: FOLLOWER
I20260927 02:47:21.953104 2019 consensus_queue.cc:260] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.953190 2019 ts_tablet_manager.cc:1428] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260927 02:47:21.953265 1991 tablet.cc:2369] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:47:21.953311 2022 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.953394 2022 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:21.973425 2022 raft_consensus.cc:482] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:21.973524 2022 raft_consensus.cc:504] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.973852 2022 leader_election.cc:288] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009831c346cc4bd8ab173a710f601a14 (127.0.98.1:35441), 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791)
I20260927 02:47:21.976861 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.974184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43497 (local address 127.0.98.3:38791)
0927 02:47:21.974241 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.974243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.974368 (+ 125us) server_negotiation.cc:407] Connection header received
0927 02:47:21.974435 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.974436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.974442 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.974477 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.974581 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.975110 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.976076 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.976554 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.976610 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.976620 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.976765 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.976799 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:47:21.976892 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.974053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32783 (local address 127.0.98.1:35441)
0927 02:47:21.974104 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.974106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.974236 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:47:21.974312 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.974313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.974321 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.974366 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:21.974526 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.975044 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.976079 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.976594 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.976651 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.976660 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.976791 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.976828 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:47:21.976917 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.974014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38791 (local address 127.0.98.2:43497)
0927 02:47:21.974326 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.974333 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.974388 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.974479 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.974480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.974481 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.974529 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.974530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.975179 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.975181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.975992 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.975995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.976619 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.976620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.976702 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.976706 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.976750 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.976788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260927 02:47:21.976923 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.974009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35441 (local address 127.0.98.2:32783)
0927 02:47:21.974172 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.974181 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.974249 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.974394 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.974395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.974397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:21.974472 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.974474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.975109 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.975111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.975994 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.975997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.976659 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.976660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.976729 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.976733 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.976777 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.976823 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":68,"thread_start_us":116,"threads_started":1}
I20260927 02:47:21.977165 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe912edabad494f9ddee26532234513" candidate_uuid: "1e44fff5969f456c802955b53039222f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009831c346cc4bd8ab173a710f601a14" is_pre_election: true
I20260927 02:47:21.977305 1662 raft_consensus.cc:2454] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e44fff5969f456c802955b53039222f in term 0.
I20260927 02:47:21.977272 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe912edabad494f9ddee26532234513" candidate_uuid: "1e44fff5969f456c802955b53039222f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996750496d941f8bc1743b398fc22bb" is_pre_election: true
I20260927 02:47:21.977361 1944 raft_consensus.cc:2454] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e44fff5969f456c802955b53039222f in term 0.
I20260927 02:47:21.977495 1736 leader_election.cc:302] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009831c346cc4bd8ab173a710f601a14, 1e44fff5969f456c802955b53039222f; no voters:
I20260927 02:47:21.977659 2022 raft_consensus.cc:2781] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:21.977690 2022 raft_consensus.cc:482] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:21.977710 2022 raft_consensus.cc:3037] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.978356 2022 raft_consensus.cc:504] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.978487 2022 leader_election.cc:288] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [CANDIDATE]: Term 1 election: Requested vote from peers 009831c346cc4bd8ab173a710f601a14 (127.0.98.1:35441), 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791)
I20260927 02:47:21.978621 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe912edabad494f9ddee26532234513" candidate_uuid: "1e44fff5969f456c802955b53039222f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6996750496d941f8bc1743b398fc22bb"
I20260927 02:47:21.978608 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe912edabad494f9ddee26532234513" candidate_uuid: "1e44fff5969f456c802955b53039222f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009831c346cc4bd8ab173a710f601a14"
I20260927 02:47:21.978683 1944 raft_consensus.cc:3037] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.978684 1662 raft_consensus.cc:3037] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:21.979382 1944 raft_consensus.cc:2454] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e44fff5969f456c802955b53039222f in term 1.
I20260927 02:47:21.979406 1662 raft_consensus.cc:2454] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e44fff5969f456c802955b53039222f in term 1.
I20260927 02:47:21.979532 1736 leader_election.cc:302] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009831c346cc4bd8ab173a710f601a14, 1e44fff5969f456c802955b53039222f; no voters:
I20260927 02:47:21.979632 2022 raft_consensus.cc:2781] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:21.979813 2022 raft_consensus.cc:686] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [term 1 LEADER]: Becoming Leader. State: Replica: 1e44fff5969f456c802955b53039222f, State: Running, Role: LEADER
I20260927 02:47:21.979887 2022 consensus_queue.cc:237] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } }
I20260927 02:47:21.980206 2056 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: TxnStatusTablet state changed. Reason: New leader 1e44fff5969f456c802955b53039222f. Latest consensus state: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.980304 2056 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:47:21.980495 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:47:21.980532 1503 catalog_manager.cc:5495] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e44fff5969f456c802955b53039222f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:21.980623 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:47:21.989712 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.986564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33984 (local address 127.0.98.2:43545)
0927 02:47:21.986604 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.986606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.986614 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:21.986644 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.986645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.986663 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.986731 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:21.986935 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.987674 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.988777 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.989072 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.989146 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.989330 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:21.989459 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:21.989525 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.989589 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.989611 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:47:21.989720 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.986241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43545 (local address 127.0.0.1:33984)
0927 02:47:21.986311 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.986323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.986382 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.986786 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.986787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.986789 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:21.986860 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.986863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.987765 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.987766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.988694 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.988697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.989153 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.989154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.989212 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.989230 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:21.989520 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:21.989528 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.989583 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.989643 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:47:21.991010 1662 raft_consensus.cc:1261] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Refusing update from remote peer 1e44fff5969f456c802955b53039222f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:21.991019 1944 raft_consensus.cc:1261] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Refusing update from remote peer 1e44fff5969f456c802955b53039222f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:21.991251 2022 consensus_queue.cc:1035] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:21.991369 2056 consensus_queue.cc:1035] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f [LEADER]: Connected to new peer: Peer: permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:21.991866 2020 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: TxnStatusTablet state changed. Reason: New leader 1e44fff5969f456c802955b53039222f. Latest consensus state: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.991992 2020 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:21.992071 2033 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: TxnStatusTablet state changed. Reason: New leader 1e44fff5969f456c802955b53039222f. Latest consensus state: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.992151 2033 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:21.992955 2022 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.993044 2022 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:47:21.993454 2022 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.993527 2022 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 1e44fff5969f456c802955b53039222f: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:47:21.993808 2033 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.993889 2033 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 009831c346cc4bd8ab173a710f601a14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:21.994136 2020 tablet_replica.cc:439] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e44fff5969f456c802955b53039222f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009831c346cc4bd8ab173a710f601a14" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35441 } } peers { permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 } } peers { permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 } } }
I20260927 02:47:21.994223 2020 tablet_replica.cc:442] T 9fe912edabad494f9ddee26532234513 P 6996750496d941f8bc1743b398fc22bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:21.998637 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.996075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43545 (local address 127.0.98.1:48361)
0927 02:47:21.996218 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:21.996228 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:21.996280 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:21.996414 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:21.996415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:21.996416 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:21.996456 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.996458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.997302 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.997302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.997871 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:21.997873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.998310 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.998310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:21.998347 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.998360 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:21.998514 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:21.998515 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:47:21.998544 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 02:47:21.998582 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260927 02:47:21.998637 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:21.996144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48361 (local address 127.0.98.2:43545)
0927 02:47:21.996186 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:21.996188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:21.996318 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:47:21.996339 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:21.996340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:21.996362 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:21.996411 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:21.996495 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.997218 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.997941 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:21.998238 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:21.998315 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:21.998435 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:21.998469 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:21.998513 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:21.998549 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 02:47:21.998568 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:47:22.000411 1944 raft_consensus.cc:1261] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Refusing update from remote peer 009831c346cc4bd8ab173a710f601a14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:22.000422 1803 raft_consensus.cc:1261] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Refusing update from remote peer 009831c346cc4bd8ab173a710f601a14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:22.000612 2025 consensus_queue.cc:1035] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e44fff5969f456c802955b53039222f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:22.000699 2033 consensus_queue.cc:1035] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6996750496d941f8bc1743b398fc22bb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:22.008767 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:22.005794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41951 (local address 127.0.98.1:35441)
0927 02:47:22.005832 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:22.005834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:22.005844 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:47:22.005909 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:22.005910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:22.005919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:22.005976 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:22.006097 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.006949 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.008123 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.008373 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.008430 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:22.008596 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:22.008620 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:22.008650 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:22.008700 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 02:47:22.008717 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:47:22.008836 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:22.005633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35441 (local address 127.0.98.2:41951)
0927 02:47:22.005801 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:22.005812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:22.005858 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:22.005979 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:22.005979 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:22.005980 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:22.006041 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:22.006042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.007035 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.007036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:22.008050 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:22.008053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.008441 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.008442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:22.008505 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:22.008522 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:22.008653 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:22.008654 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:47:22.008699 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:47:22.008737 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260927 02:47:22.016785 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:22.016883 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:22.017452 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4fc4e00c2d94e6b8b9a916b52e26981: Sending DeleteTablet for 3 replicas of tablet bfcfcc3fb26e4506bb460697570bdc75
I20260927 02:47:22.017715 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet bfcfcc3fb26e4506bb460697570bdc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48860
I20260927 02:47:22.017707 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet bfcfcc3fb26e4506bb460697570bdc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49414
I20260927 02:47:22.017829 2038 tablet_replica.cc:330] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: stopping tablet replica
I20260927 02:47:22.017885 2038 raft_consensus.cc:2229] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:22.017892 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet bfcfcc3fb26e4506bb460697570bdc75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33974
I20260927 02:47:22.017943 2038 raft_consensus.cc:2258] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:22.017987 2040 tablet_replica.cc:330] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: stopping tablet replica
I20260927 02:47:22.018010 2039 tablet_replica.cc:330] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: stopping tablet replica
I20260927 02:47:22.018038 2040 raft_consensus.cc:2229] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:22.018060 2039 raft_consensus.cc:2229] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:22.018071 2040 raft_consensus.cc:2258] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:22.018108 2039 raft_consensus.cc:2258] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:22.018493 2040 ts_tablet_manager.cc:1905] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:22.018497 2038 ts_tablet_manager.cc:1905] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:22.018844 2039 ts_tablet_manager.cc:1905] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:22.020284 2038 ts_tablet_manager.cc:1918] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:22.020287 2040 ts_tablet_manager.cc:1918] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:22.020304 2039 ts_tablet_manager.cc:1918] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:22.020339 2040 log.cc:1196] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/bfcfcc3fb26e4506bb460697570bdc75
I20260927 02:47:22.020340 2038 log.cc:1196] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/bfcfcc3fb26e4506bb460697570bdc75
I20260927 02:47:22.020345 2039 log.cc:1196] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/bfcfcc3fb26e4506bb460697570bdc75
I20260927 02:47:22.020558 2039 ts_tablet_manager.cc:1939] T bfcfcc3fb26e4506bb460697570bdc75 P 009831c346cc4bd8ab173a710f601a14: Deleting consensus metadata
I20260927 02:47:22.020561 2040 ts_tablet_manager.cc:1939] T bfcfcc3fb26e4506bb460697570bdc75 P 1e44fff5969f456c802955b53039222f: Deleting consensus metadata
I20260927 02:47:22.020569 2038 ts_tablet_manager.cc:1939] T bfcfcc3fb26e4506bb460697570bdc75 P 6996750496d941f8bc1743b398fc22bb: Deleting consensus metadata
I20260927 02:47:22.020871 1491 catalog_manager.cc:4841] TS 1e44fff5969f456c802955b53039222f (127.0.98.2:43545): tablet bfcfcc3fb26e4506bb460697570bdc75 (table test-table [id=3b3d589ab6104fdb8cd7864be7b557ef]) successfully deleted
I20260927 02:47:22.020987 1490 catalog_manager.cc:4841] TS 6996750496d941f8bc1743b398fc22bb (127.0.98.3:38791): tablet bfcfcc3fb26e4506bb460697570bdc75 (table test-table [id=3b3d589ab6104fdb8cd7864be7b557ef]) successfully deleted
I20260927 02:47:22.021014 1491 catalog_manager.cc:4841] TS 009831c346cc4bd8ab173a710f601a14 (127.0.98.1:35441): tablet bfcfcc3fb26e4506bb460697570bdc75 (table test-table [id=3b3d589ab6104fdb8cd7864be7b557ef]) successfully deleted
Sep 27 02:47:22 dist-test-slave-f4rg krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:22.028800 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:22.020849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48866 (local address 127.0.98.1:35441)
0927 02:47:22.020887 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:22.020889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:22.021207 (+ 318us) server_negotiation.cc:407] Connection header received
0927 02:47:22.021289 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:22.021290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:22.021302 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:22.021347 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:22.021857 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.022315 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.023070 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.023313 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.023361 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:22.026174 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:22.026183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:22.026184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:22.026190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:22.027753 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:22.028116 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:22.028119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:22.028119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:22.028169 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:22.028343 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:22.028344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:22.028344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:22.028477 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:22.028575 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:22.028667 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:47:22.028723 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 02:47:22.029107 1698 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48866
I20260927 02:47:22.035763 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:22.030005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.0.98.62:32789)
0927 02:47:22.030056 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:22.030058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:22.030066 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:22.030125 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:22.030126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:22.030136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:22.030188 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:22.030719 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.031296 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.032001 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:22.032208 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:22.032261 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:22.033337 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:22.033339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:22.033339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:22.033347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:22.034732 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:22.035090 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:22.035091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:22.035092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:22.035151 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:22.035307 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:22.035307 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:22.035308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:22.035417 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:22.035502 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:22.035591 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:47:22.035668 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 02:47:22.036073 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50206
I20260927 02:47:22.036659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 1576
I20260927 02:47:22.043254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 1717
I20260927 02:47:22.047757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 1858
I20260927 02:47:22.051317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 1472
2026-09-27T02:47:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6338 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:22 dist-test-slave-f4rg krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:22 dist-test-slave-f4rg krb5kdc[2078](info): set up 2 sockets
Sep 27 02:47:22 dist-test-slave-f4rg krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260927 02:47:23.002360 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:23.111454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36677
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:23.188972 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:23.189196 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:23.189227 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:23.190402 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:23.190445 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:47:23.190462 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:23.190479 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:23.190495 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:23.190510 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:23.192026 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36677
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:23.192283 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:23.192488 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:23.197297 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:23.197410 2091 server_base.cc:1034] running on GCE node
W20260927 02:47:23.197305 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:23.197316 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:23.197791 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:23.198104 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:23.199271 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790477243199255 us; error 38 us; skew 500 ppm
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:23.201685 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:23.202128 2091 webserver.cc:460] Webserver started at http://127.0.98.62:32889/ using document root <none> and password file <none>
I20260927 02:47:23.202337 2091 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:23.202376 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:23.202469 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:23.203524 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "8de085e7e7ce4c938c630040d521eeaa"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "bb9cde327798ef5f15f2f77ceb083cbd"
server_key_iv: "cb83d76eb9752f98c77954572bdb0d49"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.203821 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "8de085e7e7ce4c938c630040d521eeaa"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "bb9cde327798ef5f15f2f77ceb083cbd"
server_key_iv: "cb83d76eb9752f98c77954572bdb0d49"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.205154 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:23.205951 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.206125 2091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:23.206213 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "8de085e7e7ce4c938c630040d521eeaa"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "bb9cde327798ef5f15f2f77ceb083cbd"
server_key_iv: "cb83d76eb9752f98c77954572bdb0d49"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.206274 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:23.226406 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:23.228551 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:23.228669 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:23.233189 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36677
I20260927 02:47:23.233243 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36677 every 8 connection(s)
I20260927 02:47:23.233582 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:23.234196 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:23.235628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 2091
I20260927 02:47:23.235702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:23.235844 392 external_mini_cluster.cc:1428] Setting key 91b6f4185db2c5753fd8dd56c1221697
I20260927 02:47:23.236703 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Bootstrap starting.
I20260927 02:47:23.237257 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:23.237519 2169 log.cc:824] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:23.238250 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: No bootstrap required, opened a new log
I20260927 02:47:23.240483 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de085e7e7ce4c938c630040d521eeaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36677 } }
I20260927 02:47:23.240665 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:23.240743 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de085e7e7ce4c938c630040d521eeaa, State: Initialized, Role: FOLLOWER
I20260927 02:47:23.240922 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de085e7e7ce4c938c630040d521eeaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36677 } }
I20260927 02:47:23.241008 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:23.241040 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:23.241088 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:23.241649 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de085e7e7ce4c938c630040d521eeaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36677 } }
I20260927 02:47:23.241748 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8de085e7e7ce4c938c630040d521eeaa; no voters:
I20260927 02:47:23.241928 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:23.241983 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:23.242179 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:23.242190 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [term 1 LEADER]: Becoming Leader. State: Replica: 8de085e7e7ce4c938c630040d521eeaa, State: Running, Role: LEADER
I20260927 02:47:23.242388 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de085e7e7ce4c938c630040d521eeaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36677 } }
I20260927 02:47:23.242753 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8de085e7e7ce4c938c630040d521eeaa. Latest consensus state: current_term: 1 leader_uuid: "8de085e7e7ce4c938c630040d521eeaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de085e7e7ce4c938c630040d521eeaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36677 } } }
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:23.242764 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8de085e7e7ce4c938c630040d521eeaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de085e7e7ce4c938c630040d521eeaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36677 } } }
I20260927 02:47:23.242950 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:23.242961 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:23.243296 2183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:23.243867 2183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:23.245869 2183 catalog_manager.cc:1347] Generated new cluster ID: 8e0c02c63e964ddb940c85c439575846
I20260927 02:47:23.245924 2183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:23.247030 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.236375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40558 (local address 127.0.98.62:36677)
0927 02:47:23.236544 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.236552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.236577 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:47:23.237016 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.237024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.237087 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.237215 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.237866 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.239262 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.240022 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.240640 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.240769 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.243411 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.243415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.243416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.243429 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.245416 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.245764 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.245768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.245769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.245873 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.246091 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.246093 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.246093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.246399 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.246542 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.246638 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.246778 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260927 02:47:23.249749 2183 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:23.250238 2183 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:23.252700 2183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Generated new TSK 0
I20260927 02:47:23.252854 2183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:23.283022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36677
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:23.396341 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:23.396546 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:23.396571 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:23.397744 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:23.397782 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:23.397802 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:47:23.397814 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:47:23.397830 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:23.399318 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:23.399547 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:23.399765 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:23.405210 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:23.405344 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:23.405202 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:23.405450 2195 server_base.cc:1034] running on GCE node
I20260927 02:47:23.405629 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:23.405922 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:23.407074 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790477243407056 us; error 31 us; skew 500 ppm
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:23.409876 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:23.410346 2195 webserver.cc:460] Webserver started at http://127.0.98.1:46197/ using document root <none> and password file <none>
I20260927 02:47:23.410535 2195 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:23.410573 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:23.410663 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:23.411784 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "2ce04d15fb77485293b97b7712700612"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "5942188bef0aa5c9df119714b43ee61f"
server_key_iv: "31e848f65559bb50c20211aa9ac1ad6a"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.412074 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2ce04d15fb77485293b97b7712700612"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "5942188bef0aa5c9df119714b43ee61f"
server_key_iv: "31e848f65559bb50c20211aa9ac1ad6a"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.413225 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:23.413903 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.414043 2195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:23.414124 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "2ce04d15fb77485293b97b7712700612"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "5942188bef0aa5c9df119714b43ee61f"
server_key_iv: "31e848f65559bb50c20211aa9ac1ad6a"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.414189 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:23.431284 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:23.433363 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:23.433506 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:23.434010 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:23.434047 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.434082 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:23.434104 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.440462 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36017
I20260927 02:47:23.440546 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36017 every 8 connection(s)
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:23.440853 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:23.443815 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.434533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57387 (local address 127.0.98.62:36677)
0927 02:47:23.434581 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.434583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.434842 (+ 259us) server_negotiation.cc:407] Connection header received
0927 02:47:23.435397 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.435399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.435413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.435477 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.436349 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.437117 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.438585 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.438976 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.439049 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.441307 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.441308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.441308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.441315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.442626 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.442986 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.442987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.442987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.443033 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.443199 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.443199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.443200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.443294 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.443346 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.443621 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.443696 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:47:23.443871 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.434538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36677 (local address 127.0.98.1:57387)
0927 02:47:23.434703 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.434738 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.435279 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.435573 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.435578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.435967 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:23.436249 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.436255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.437215 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.437216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.438495 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.438499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.439074 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.439075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.439179 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.439677 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:23.439691 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:23.441212 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:23.442686 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.442691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.442693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.442938 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.443077 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.443078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.443078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.443165 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.443371 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:23.443377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:23.443502 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.443586 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.443704 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260927 02:47:23.444294 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36677
I20260927 02:47:23.444372 2326 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:23.444532 2326 heartbeater.cc:511] Master 127.0.98.62:36677 requested a full tablet report, sending...
I20260927 02:47:23.445060 2122 ts_manager.cc:194] Registered new tserver with Master: 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017)
I20260927 02:47:23.445708 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57387
I20260927 02:47:23.448556 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 2195
I20260927 02:47:23.448655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:23.448829 392 external_mini_cluster.cc:1428] Setting key 736832a1c5208fe3f53bbd3e9e14cc35
I20260927 02:47:23.450181 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.445321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58995 (local address 127.0.98.62:36677)
0927 02:47:23.445374 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.445375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.445384 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:23.445444 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.445445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.445454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.445497 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.445760 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.446288 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.446975 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.447163 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.447215 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.447945 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.447946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.447947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.447953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.449212 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.449524 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.449525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.449525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.449575 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.449744 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.449745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.449745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.449847 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.449898 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.450029 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.450091 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:47:23.450194 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.445193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36677 (local address 127.0.98.1:58995)
0927 02:47:23.445326 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.445335 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.445400 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.445500 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.445501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.445655 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:23.445699 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.445700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.446373 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.446374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.446911 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.446912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.447217 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.447218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.447286 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.447625 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:23.447629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:23.447874 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:23.449274 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.449275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.449275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.449469 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.449632 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.449633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.449633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.449696 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.449898 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:23.449900 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:23.449942 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.449999 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.450093 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:23.470870 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36677
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:23.549199 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:23.549414 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:23.549446 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:23.550593 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:23.550634 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:23.550662 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:47:23.550676 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:47:23.550709 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:23.552191 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:23.552426 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:23.552646 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:23.557582 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:23.557590 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:23.557590 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:23.557751 2335 server_base.cc:1034] running on GCE node
I20260927 02:47:23.557950 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:23.558175 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:23.559325 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790477243559306 us; error 29 us; skew 500 ppm
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:23.561733 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:23.562199 2335 webserver.cc:460] Webserver started at http://127.0.98.2:35313/ using document root <none> and password file <none>
I20260927 02:47:23.562391 2335 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:23.562443 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:23.562566 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:23.563634 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "116a4a35293e43e4b3bd4d76c55d5f3f"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "e6c752816fb4deb3213607350eb9ae06"
server_key_iv: "7ef07d65837a38a1aaac753328ef868e"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.563970 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "116a4a35293e43e4b3bd4d76c55d5f3f"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "e6c752816fb4deb3213607350eb9ae06"
server_key_iv: "7ef07d65837a38a1aaac753328ef868e"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.565204 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:47:23.565845 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.565992 2335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:23.566071 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "116a4a35293e43e4b3bd4d76c55d5f3f"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "e6c752816fb4deb3213607350eb9ae06"
server_key_iv: "7ef07d65837a38a1aaac753328ef868e"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.566136 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:23.580747 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:23.582968 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:23.583102 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:23.583618 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:23.583659 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.583689 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:23.583705 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:23.590287 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38401
I20260927 02:47:23.590413 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38401 every 8 connection(s)
I20260927 02:47:23.590683 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:23.592742 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.584023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38943 (local address 127.0.98.62:36677)
0927 02:47:23.584087 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.584089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.584264 (+ 175us) server_negotiation.cc:407] Connection header received
0927 02:47:23.584736 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.584737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.584749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.584821 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.585587 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.586325 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.587368 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.587661 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.587738 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.590053 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.590054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.590055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.590063 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.591525 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.591875 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.591876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.591877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.591939 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.592147 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.592148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.592148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.592255 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.592307 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.592581 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.592656 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:47:23.592901 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.583996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36677 (local address 127.0.98.2:38943)
0927 02:47:23.584145 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.584185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.584648 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.584890 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.584894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.585196 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:23.585490 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.585496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.586416 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.586417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.587298 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.587300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.587754 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.587755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.587879 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.588353 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:23.588365 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:23.589959 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:23.591598 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.591604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.591606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.591807 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.591991 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.591992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.591992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.592093 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.592307 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:23.592313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:23.592459 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.592553 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.592703 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260927 02:47:23.593317 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36677
I20260927 02:47:23.593395 2466 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:23.593555 2466 heartbeater.cc:511] Master 127.0.98.62:36677 requested a full tablet report, sending...
I20260927 02:47:23.594216 2123 ts_manager.cc:194] Registered new tserver with Master: 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401)
I20260927 02:47:23.594601 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38943
I20260927 02:47:23.595238 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 2335
I20260927 02:47:23.595326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:23.595523 392 external_mini_cluster.cc:1428] Setting key cced78ab459ef4990b1c2d1f2493842c
I20260927 02:47:23.599716 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.594303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51613 (local address 127.0.98.62:36677)
0927 02:47:23.594334 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.594336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.594794 (+ 458us) server_negotiation.cc:407] Connection header received
0927 02:47:23.594883 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.594884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.594894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.594956 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.595242 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.595693 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.596401 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.596647 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.596708 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.597471 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.597472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.597473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.597479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.598650 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.599010 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.599010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.599011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.599056 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.599237 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.599237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.599238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.599388 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.599438 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.599597 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.599658 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:47:23.599779 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.594251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36677 (local address 127.0.98.2:51613)
0927 02:47:23.594731 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.594742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.594836 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.594961 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.594961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.595124 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:23.595176 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.595177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.595768 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.595769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.596329 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.596331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.596709 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.596709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.596763 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.597123 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:23.597126 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:23.597382 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:23.598728 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.598728 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.598729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.598946 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.599112 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.599113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.599113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.599180 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.599455 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:23.599456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:23.599505 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.599580 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.599679 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:23.617550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36677
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:23.695780 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:23.695995 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:23.696028 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:23.697156 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:23.697198 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:23.697225 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 02:47:23.697239 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:47:23.697261 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:23.698829 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:23.699093 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:23.699340 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:23.703836 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:23.703852 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:23.703936 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:23.704128 2475 server_base.cc:1034] running on GCE node
I20260927 02:47:23.704303 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:23.704516 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:23.705647 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790477243705638 us; error 22 us; skew 500 ppm
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:23.708041 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:23.708474 2475 webserver.cc:460] Webserver started at http://127.0.98.3:33787/ using document root <none> and password file <none>
I20260927 02:47:23.708662 2475 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:23.708701 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:23.708794 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:23.709800 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "c0705588e87d45a6978d68ecece2a877"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "7222f86602a06d7a28fb1c2a681b2b04"
server_key_iv: "e7ab97735910477a112de9a0778fabf4"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.710104 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c0705588e87d45a6978d68ecece2a877"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "7222f86602a06d7a28fb1c2a681b2b04"
server_key_iv: "e7ab97735910477a112de9a0778fabf4"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.711310 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 02:47:23.712009 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.712164 2475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.712235 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "c0705588e87d45a6978d68ecece2a877"
format_stamp: "Formatted at 2026-09-27 02:47:23 on dist-test-slave-f4rg"
server_key: "7222f86602a06d7a28fb1c2a681b2b04"
server_key_iv: "e7ab97735910477a112de9a0778fabf4"
server_key_version: "encryptionkey@0"
I20260927 02:47:23.712304 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:23.755869 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:23.757745 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:23.757853 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:23.758304 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:23.758342 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.758364 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:23.758378 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:23.764628 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40409
I20260927 02:47:23.764706 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40409 every 8 connection(s)
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:23.765008 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:23.767763 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.758838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48743 (local address 127.0.98.62:36677)
0927 02:47:23.758877 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.758878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.759111 (+ 233us) server_negotiation.cc:407] Connection header received
0927 02:47:23.759639 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.759641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.759656 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.759724 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.760533 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.761280 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.762794 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.763111 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.763192 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.765370 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.765371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.765372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.765380 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.766660 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.767056 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.767056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.767057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.767104 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.767265 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.767266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.767266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.767357 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.767398 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.767601 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.767655 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:47:23.767874 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.758787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36677 (local address 127.0.98.3:48743)
0927 02:47:23.758991 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.759026 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.759522 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.759794 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.759798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.760140 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:23.760436 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.760442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.761394 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.761395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.762711 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.762713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.763206 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.763207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.763309 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.763816 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:23.763827 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:23.765267 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:23.766733 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.766736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.766738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.766993 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.767141 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.767142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.767142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.767228 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.767399 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:23.767407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:23.767515 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.767590 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.767702 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
I20260927 02:47:23.768290 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36677
I20260927 02:47:23.768385 2606 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:23.768556 2606 heartbeater.cc:511] Master 127.0.98.62:36677 requested a full tablet report, sending...
I20260927 02:47:23.768929 2122 ts_manager.cc:194] Registered new tserver with Master: c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409)
I20260927 02:47:23.769364 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48743
I20260927 02:47:23.772373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 2475
I20260927 02:47:23.772465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:23.772646 392 external_mini_cluster.cc:1428] Setting key 5808d24c288a475002d136004231012e
I20260927 02:47:23.773324 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:47:23.774314 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.769256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54389 (local address 127.0.98.62:36677)
0927 02:47:23.769296 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.769298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.769335 (+ 37us) server_negotiation.cc:407] Connection header received
0927 02:47:23.769418 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.769419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.769428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.769483 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.769874 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.770327 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.771053 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.771241 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.771289 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.772031 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.772032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.772032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.772039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.773348 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.773683 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.773683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.773683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.773774 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.773941 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.773941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.773942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.774029 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.774069 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.774208 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.774258 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:47:23.774371 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.769168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36677 (local address 127.0.98.3:54389)
0927 02:47:23.769286 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.769296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.769358 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.769528 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.769529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.769733 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:23.769796 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.769797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.770409 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.770410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.770979 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.770981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.771293 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.771294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.771361 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.771694 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:23.771697 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:23.771957 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:23.773434 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.773435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.773436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.773623 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.773827 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:23.773828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:23.773828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:23.773887 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:23.774078 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:23.774079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:23.774119 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.774179 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.774274 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:47:23 dist-test-slave-f4rg krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:23.794629 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.787971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40572 (local address 127.0.98.62:36677)
0927 02:47:23.788016 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.788018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.788026 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:23.788081 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.788082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.788091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.788138 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:23.788656 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.789185 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.789878 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.790059 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.790112 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.792204 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:23.792206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:23.792206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:23.792213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:23.793563 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.793892 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.793893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.793893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.793951 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:23.794165 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:23.794166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:23.794166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:23.794309 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:23.794387 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.794499 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.794559 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:47:23.796064 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:23.797590 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:23.803787 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.800043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.0.98.1:36017)
0927 02:47:23.800205 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.800211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.800219 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:23.800251 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.800252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.800265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.800333 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:23.800505 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.801372 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.802746 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.803411 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.803471 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.803531 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.803651 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.803694 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260927 02:47:23.803781 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.799787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36017 (local address 127.0.0.1:52740)
0927 02:47:23.799947 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.799985 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.800049 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.800363 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.800367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.800369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:23.800435 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.800436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.801443 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.801444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.802646 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.802649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.803480 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.803481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.803554 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.803566 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.803631 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.803684 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":56,"threads_started":1}
I20260927 02:47:23.803965 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.799957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.0.98.3:40409)
0927 02:47:23.800258 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.800262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.800354 (+ 92us) server_negotiation.cc:407] Connection header received
0927 02:47:23.800396 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.800398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.800421 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.800493 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:23.800679 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.801813 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.802857 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.803454 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.803519 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.803570 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.803823 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.803864 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":79,"threads_started":1}
I20260927 02:47:23.804003 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.799788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40409 (local address 127.0.0.1:46406)
0927 02:47:23.800289 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.800295 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.800337 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.800555 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.800556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.800558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:23.800608 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.800609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.801908 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.801909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.802761 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.802764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.803603 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.803605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.803666 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.803670 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.803714 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.803753 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":35,"thread_start_us":74,"threads_started":1}
I20260927 02:47:23.804178 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.799788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.0.1:60160)
0927 02:47:23.800115 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:23.800124 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:23.800180 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:23.800629 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:23.800630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:23.800632 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:23.800666 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.800667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.801899 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.801900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.803074 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:23.803076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.803969 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.803970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:23.804029 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.804033 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:23.804081 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:47:23.804119 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":26,"thread_start_us":93,"threads_started":1}
I20260927 02:47:23.804415 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:23.800024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.0.98.2:38401)
0927 02:47:23.800450 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:23.800454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:23.800462 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:23.800494 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:23.800496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:23.800510 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:23.800571 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:23.800731 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.801825 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.803152 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:23.803827 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:23.803882 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:23.803935 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:23.804278 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 02:47:23.804316 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":75,"threads_started":1}
I20260927 02:47:23.804864 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 741a769e0495470081e5e05f625ae184 (DEFAULT_TABLE table=test-table [id=a1a045f35e7a4a3a8774a269385aae0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:23.804956 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 741a769e0495470081e5e05f625ae184 (DEFAULT_TABLE table=test-table [id=a1a045f35e7a4a3a8774a269385aae0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:23.804958 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 741a769e0495470081e5e05f625ae184 (DEFAULT_TABLE table=test-table [id=a1a045f35e7a4a3a8774a269385aae0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:23.805156 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 741a769e0495470081e5e05f625ae184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:23.805272 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 741a769e0495470081e5e05f625ae184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:23.805272 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 741a769e0495470081e5e05f625ae184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:23.807500 2629 tablet_bootstrap.cc:492] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Bootstrap starting.
I20260927 02:47:23.808303 2629 tablet_bootstrap.cc:654] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:23.808555 2629 log.cc:824] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:23.808929 2630 tablet_bootstrap.cc:492] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Bootstrap starting.
I20260927 02:47:23.809495 2630 tablet_bootstrap.cc:654] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:23.809528 2629 tablet_bootstrap.cc:492] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: No bootstrap required, opened a new log
I20260927 02:47:23.809585 2629 ts_tablet_manager.cc:1397] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:23.809773 2630 log.cc:824] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:23.810075 2631 tablet_bootstrap.cc:492] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Bootstrap starting.
I20260927 02:47:23.810389 2630 tablet_bootstrap.cc:492] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: No bootstrap required, opened a new log
I20260927 02:47:23.810434 2630 ts_tablet_manager.cc:1397] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:23.811429 2629 raft_consensus.cc:348] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:23.811417 2631 tablet_bootstrap.cc:654] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:23.811595 2629 raft_consensus.cc:374] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:23.811626 2629 raft_consensus.cc:729] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0705588e87d45a6978d68ecece2a877, State: Initialized, Role: FOLLOWER
I20260927 02:47:23.811745 2629 consensus_queue.cc:260] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:23.811918 2631 log.cc:824] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:23.812006 2629 ts_tablet_manager.cc:1428] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 02:47:23.812059 2606 heartbeater.cc:503] Master 127.0.98.62:36677 was elected leader, sending a full tablet report...
I20260927 02:47:23.812014 2630 raft_consensus.cc:348] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:23.812456 2630 raft_consensus.cc:374] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:23.812494 2630 raft_consensus.cc:729] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116a4a35293e43e4b3bd4d76c55d5f3f, State: Initialized, Role: FOLLOWER
I20260927 02:47:23.812585 2631 tablet_bootstrap.cc:492] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: No bootstrap required, opened a new log
I20260927 02:47:23.812594 2630 consensus_queue.cc:260] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:23.812637 2631 ts_tablet_manager.cc:1397] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:47:23.812831 2630 ts_tablet_manager.cc:1428] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:23.812877 2466 heartbeater.cc:503] Master 127.0.98.62:36677 was elected leader, sending a full tablet report...
I20260927 02:47:23.814649 2631 raft_consensus.cc:348] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:23.814771 2631 raft_consensus.cc:374] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:23.814790 2631 raft_consensus.cc:729] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce04d15fb77485293b97b7712700612, State: Initialized, Role: FOLLOWER
I20260927 02:47:23.814908 2631 consensus_queue.cc:260] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:23.815106 2631 ts_tablet_manager.cc:1428] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:23.815178 2326 heartbeater.cc:503] Master 127.0.98.62:36677 was elected leader, sending a full tablet report...
W20260927 02:47:23.841251 2467 tablet.cc:2369] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:47:23.941681 2327 tablet.cc:2369] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:47:24.015578 2607 tablet.cc:2369] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:47:24.043746 2637 raft_consensus.cc:482] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:24.043911 2637 raft_consensus.cc:504] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:24.044242 2637 leader_election.cc:288] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409), 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401)
I20260927 02:47:24.047967 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.044554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38499 (local address 127.0.98.3:40409)
0927 02:47:24.044592 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.044594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.044602 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:24.044655 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.044656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.044669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.044721 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:24.044899 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.045649 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.047076 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.047630 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.047705 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.047717 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.047863 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.047898 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:47:24.047979 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.044408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40409 (local address 127.0.98.1:38499)
0927 02:47:24.044526 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.044537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.044594 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.044734 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.044735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.044738 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:24.044811 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.044813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.045721 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.045722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.046984 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.046987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.047710 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.047711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.047792 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.047796 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.047848 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.047890 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":68,"thread_start_us":57,"threads_started":1}
I20260927 02:47:24.048002 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.044506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51873 (local address 127.0.98.2:38401)
0927 02:47:24.044554 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.044556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.044631 (+ 75us) server_negotiation.cc:407] Connection header received
0927 02:47:24.044677 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.044678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.044740 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.044791 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:24.044950 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.045649 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.047075 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.047641 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.047766 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.047776 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.047913 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.047949 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:47:24.047984 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.044413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.98.1:51873)
0927 02:47:24.044572 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.044580 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.044630 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.044805 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.044806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.044807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:24.044864 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.044865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.045724 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.045725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.046984 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.046987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.047777 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.047778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.047851 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.047854 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.047897 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.047923 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":20,"thread_start_us":64,"threads_started":1}
I20260927 02:47:24.048381 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741a769e0495470081e5e05f625ae184" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" is_pre_election: true
I20260927 02:47:24.048538 2420 raft_consensus.cc:2454] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 0.
I20260927 02:47:24.048761 2212 leader_election.cc:302] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, 2ce04d15fb77485293b97b7712700612; no voters:
I20260927 02:47:24.048799 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741a769e0495470081e5e05f625ae184" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877" is_pre_election: true
I20260927 02:47:24.048893 2637 raft_consensus.cc:2781] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:24.048940 2637 raft_consensus.cc:482] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:24.048928 2560 raft_consensus.cc:2454] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 0.
I20260927 02:47:24.048960 2637 raft_consensus.cc:3037] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.049654 2637 raft_consensus.cc:504] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:24.049772 2637 leader_election.cc:288] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 election: Requested vote from peers c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409), 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401)
I20260927 02:47:24.049998 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741a769e0495470081e5e05f625ae184" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877"
I20260927 02:47:24.049999 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "741a769e0495470081e5e05f625ae184" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f"
I20260927 02:47:24.050091 2560 raft_consensus.cc:3037] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.050091 2420 raft_consensus.cc:3037] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.051008 2560 raft_consensus.cc:2454] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 1.
I20260927 02:47:24.051008 2420 raft_consensus.cc:2454] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 1.
I20260927 02:47:24.051185 2212 leader_election.cc:302] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, 2ce04d15fb77485293b97b7712700612; no voters:
I20260927 02:47:24.051278 2637 raft_consensus.cc:2781] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:24.051406 2637 raft_consensus.cc:686] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce04d15fb77485293b97b7712700612, State: Running, Role: LEADER
I20260927 02:47:24.051502 2637 consensus_queue.cc:237] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } }
I20260927 02:47:24.052315 2123 catalog_manager.cc:5495] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ce04d15fb77485293b97b7712700612 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2ce04d15fb77485293b97b7712700612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:47:24 dist-test-slave-f4rg krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:24.113005 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.103468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36677 (local address 127.0.0.1:40582)
0927 02:47:24.103625 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.103634 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.103701 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.103896 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.103897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.104195 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:24.104246 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.104248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.104842 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.104843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.105366 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.105368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.105704 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.105705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.105753 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.109231 (+ 3478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:24.109236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:24.110621 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:24.112082 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:24.112088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:24.112090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:24.112283 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:24.112470 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:24.112470 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:24.112471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:24.112524 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:24.112715 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:24.112719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:24.112758 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.112803 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.112916 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":16,"thread_start_us":81,"threads_started":1}
I20260927 02:47:24.113052 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.103551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.0.98.62:36677)
0927 02:47:24.103617 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.103619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.103696 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:47:24.103792 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.103793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.103802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.103855 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:24.104306 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.104781 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.105454 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.105635 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.105698 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.110712 (+ 5014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:24.110714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:24.110714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:24.110720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:24.111995 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:24.112361 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:24.112363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:24.112363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:24.112417 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:24.112571 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:24.112572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:24.112572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:24.112673 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:24.112724 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.112827 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.112958 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":41}
I20260927 02:47:24.114902 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40582:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:47:24.115182 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:24.116554 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 672b571fa9464ccfa9c35c2cf64d890d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58cd391c59a44c299946b65d867d8906]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:47:24.116561 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 672b571fa9464ccfa9c35c2cf64d890d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58cd391c59a44c299946b65d867d8906]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:47:24.116701 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b571fa9464ccfa9c35c2cf64d890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:24.116698 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b571fa9464ccfa9c35c2cf64d890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:24.117856 2631 tablet_bootstrap.cc:492] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: Bootstrap starting.
I20260927 02:47:24.117950 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 672b571fa9464ccfa9c35c2cf64d890d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=58cd391c59a44c299946b65d867d8906]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:47:24.118010 2629 tablet_bootstrap.cc:492] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: Bootstrap starting.
I20260927 02:47:24.118100 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 672b571fa9464ccfa9c35c2cf64d890d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:24.118286 2631 tablet_bootstrap.cc:654] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:24.118489 2629 tablet_bootstrap.cc:654] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:24.118974 2631 tablet_bootstrap.cc:492] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: No bootstrap required, opened a new log
I20260927 02:47:24.119016 2631 ts_tablet_manager.cc:1397] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:24.119277 2630 tablet_bootstrap.cc:492] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: Bootstrap starting.
I20260927 02:47:24.119318 2631 raft_consensus.cc:348] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.119381 2631 raft_consensus.cc:374] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:24.119400 2631 raft_consensus.cc:729] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce04d15fb77485293b97b7712700612, State: Initialized, Role: FOLLOWER
I20260927 02:47:24.119448 2631 consensus_queue.cc:260] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.119529 2631 ts_tablet_manager.cc:1428] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:24.119607 2630 tablet_bootstrap.cc:654] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:24.119575 2637 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.119658 2637 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:24.120061 2629 tablet_bootstrap.cc:492] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: No bootstrap required, opened a new log
I20260927 02:47:24.120112 2629 ts_tablet_manager.cc:1397] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:47:24.120246 2629 raft_consensus.cc:348] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.120322 2629 raft_consensus.cc:374] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:24.120342 2629 raft_consensus.cc:729] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0705588e87d45a6978d68ecece2a877, State: Initialized, Role: FOLLOWER
I20260927 02:47:24.120388 2629 consensus_queue.cc:260] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.120478 2629 ts_tablet_manager.cc:1428] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:24.120561 2634 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.120661 2630 tablet_bootstrap.cc:492] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: No bootstrap required, opened a new log
I20260927 02:47:24.120661 2634 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:24.120698 2630 ts_tablet_manager.cc:1397] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:24.120841 2630 raft_consensus.cc:348] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.120910 2630 raft_consensus.cc:374] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:24.120934 2630 raft_consensus.cc:729] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116a4a35293e43e4b3bd4d76c55d5f3f, State: Initialized, Role: FOLLOWER
I20260927 02:47:24.120976 2630 consensus_queue.cc:260] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.121079 2630 ts_tablet_manager.cc:1428] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:24.121109 2636 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.121177 2636 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:24.188468 2634 raft_consensus.cc:482] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:24.188568 2634 raft_consensus.cc:504] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.188908 2634 leader_election.cc:288] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017), 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401)
I20260927 02:47:24.191761 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.189161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42775 (local address 127.0.98.2:38401)
0927 02:47:24.189215 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.189217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.189228 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:47:24.189276 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.189277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.189284 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.189344 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:24.189486 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.189975 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.191107 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.191462 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.191520 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.191530 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.191660 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.191691 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:47:24.191792 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.189092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.98.3:42775)
0927 02:47:24.189169 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.189180 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.189221 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.189351 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.189353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.189355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:24.189427 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.189428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.190049 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.190049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.191033 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.191035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.191527 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.191527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.191604 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.191607 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.191651 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.191690 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 02:47:24.191790 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.189188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52895 (local address 127.0.98.1:36017)
0927 02:47:24.189224 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.189225 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.189416 (+ 191us) server_negotiation.cc:407] Connection header received
0927 02:47:24.189445 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.189446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.189455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.189495 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:24.189632 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.190093 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.191136 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.191491 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.191550 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.191559 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.191688 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.191723 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:47:24.191821 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.189096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36017 (local address 127.0.98.3:52895)
0927 02:47:24.189320 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.189328 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.189371 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.189505 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.189505 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.189506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:24.189572 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.189573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.190162 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.190163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.191060 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.191063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.191576 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.191577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.191633 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.191636 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.191678 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.191716 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":69,"thread_start_us":113,"threads_started":1}
I20260927 02:47:24.192004 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b571fa9464ccfa9c35c2cf64d890d" candidate_uuid: "c0705588e87d45a6978d68ecece2a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" is_pre_election: true
I20260927 02:47:24.192101 2420 raft_consensus.cc:2454] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0705588e87d45a6978d68ecece2a877 in term 0.
I20260927 02:47:24.192088 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b571fa9464ccfa9c35c2cf64d890d" candidate_uuid: "c0705588e87d45a6978d68ecece2a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce04d15fb77485293b97b7712700612" is_pre_election: true
I20260927 02:47:24.192224 2280 raft_consensus.cc:2454] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0705588e87d45a6978d68ecece2a877 in term 0.
I20260927 02:47:24.192276 2492 leader_election.cc:302] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, c0705588e87d45a6978d68ecece2a877; no voters:
I20260927 02:47:24.192380 2634 raft_consensus.cc:2781] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:24.192410 2634 raft_consensus.cc:482] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:24.192428 2634 raft_consensus.cc:3037] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.193110 2634 raft_consensus.cc:504] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.193226 2634 leader_election.cc:288] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [CANDIDATE]: Term 1 election: Requested vote from peers 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017), 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401)
I20260927 02:47:24.193360 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b571fa9464ccfa9c35c2cf64d890d" candidate_uuid: "c0705588e87d45a6978d68ecece2a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce04d15fb77485293b97b7712700612"
I20260927 02:47:24.193365 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "672b571fa9464ccfa9c35c2cf64d890d" candidate_uuid: "c0705588e87d45a6978d68ecece2a877" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f"
I20260927 02:47:24.193430 2420 raft_consensus.cc:3037] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.193430 2280 raft_consensus.cc:3037] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.194020 2420 raft_consensus.cc:2454] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0705588e87d45a6978d68ecece2a877 in term 1.
I20260927 02:47:24.194026 2280 raft_consensus.cc:2454] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0705588e87d45a6978d68ecece2a877 in term 1.
I20260927 02:47:24.194185 2492 leader_election.cc:302] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, c0705588e87d45a6978d68ecece2a877; no voters:
I20260927 02:47:24.194286 2634 raft_consensus.cc:2781] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:24.194428 2634 raft_consensus.cc:686] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [term 1 LEADER]: Becoming Leader. State: Replica: c0705588e87d45a6978d68ecece2a877, State: Running, Role: LEADER
I20260927 02:47:24.194520 2634 consensus_queue.cc:237] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.194944 2652 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: TxnStatusTablet state changed. Reason: New leader c0705588e87d45a6978d68ecece2a877. Latest consensus state: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.195019 2652 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:47:24.195240 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:47:24.195348 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:47:24.195279 2123 catalog_manager.cc:5495] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0705588e87d45a6978d68ecece2a877 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } health_report { overall_health: HEALTHY } } }
I20260927 02:47:24.213505 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.211149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.0.98.3:40409)
0927 02:47:24.211190 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.211192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.211225 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:47:24.211334 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.211335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.211349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.211404 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:24.211535 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.211985 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.212783 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.212964 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.213018 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.213178 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:24.213271 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:24.213346 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.213384 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.213406 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:47:24.213495 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.211088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40409 (local address 127.0.0.1:46414)
0927 02:47:24.211183 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.211194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.211242 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.211421 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.211422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.211423 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:24.211477 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.211479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.212060 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.212062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.212701 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.212704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.213021 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.213022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.213086 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.213101 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:24.213323 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:24.213330 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.213377 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.213428 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 02:47:24.215447 2420 raft_consensus.cc:1261] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Refusing update from remote peer c0705588e87d45a6978d68ecece2a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:24.215482 2280 raft_consensus.cc:1261] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Refusing update from remote peer c0705588e87d45a6978d68ecece2a877: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:24.215740 2652 consensus_queue.cc:1035] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:24.215889 2634 consensus_queue.cc:1035] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:24.216926 2637 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: TxnStatusTablet state changed. Reason: New leader c0705588e87d45a6978d68ecece2a877. Latest consensus state: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.217038 2637 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:24.217085 2636 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: TxnStatusTablet state changed. Reason: New leader c0705588e87d45a6978d68ecece2a877. Latest consensus state: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.217167 2636 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:24.217994 2634 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.218070 2634 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:47:24.218146 2636 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.218225 2636 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P 116a4a35293e43e4b3bd4d76c55d5f3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:24.218386 2652 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.218448 2652 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P c0705588e87d45a6978d68ecece2a877: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:47:24.219162 2658 mvcc.cc:205] Tried to move back new op lower bound from 7333794792303632384 to 7333794792221270016. Current Snapshot: MvccSnapshot[applied={T|T < 7333794792303632384}]
I20260927 02:47:24.219257 2637 tablet_replica.cc:439] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0705588e87d45a6978d68ecece2a877" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } } }
I20260927 02:47:24.219332 2637 tablet_replica.cc:442] T 672b571fa9464ccfa9c35c2cf64d890d P 2ce04d15fb77485293b97b7712700612: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:47:24.219416 2659 mvcc.cc:205] Tried to move back new op lower bound from 7333794792303632384 to 7333794792221270016. Current Snapshot: MvccSnapshot[applied={T|T < 7333794792303632384}]
I20260927 02:47:24.225359 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.223159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.0.98.1:36017)
0927 02:47:24.223200 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.223203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.223211 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:24.223278 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.223278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.223295 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.223371 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:24.223493 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.223965 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.224677 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.224873 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.224927 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.225061 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:24.225157 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:24.225221 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.225258 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.225278 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:47:24.230020 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.227676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40409 (local address 127.0.98.1:43547)
0927 02:47:24.227854 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.227863 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.227914 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.228019 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.228020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.228031 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:24.228086 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.228087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.228672 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.228673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.229265 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.229267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.229571 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.229573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.229632 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.229646 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:24.229824 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:24.229842 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.229883 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.229925 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
I20260927 02:47:24.230020 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.227738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43547 (local address 127.0.98.3:40409)
0927 02:47:24.227781 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.227784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.227913 (+ 129us) server_negotiation.cc:407] Connection header received
0927 02:47:24.227959 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.227960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.227970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.228010 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:24.228134 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.228604 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.229340 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.229517 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.229568 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.229737 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:24.229768 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:24.229807 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.229924 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.229947 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:47:24.232239 2420 raft_consensus.cc:1261] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Refusing update from remote peer 2ce04d15fb77485293b97b7712700612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:24.232304 2560 raft_consensus.cc:1261] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Refusing update from remote peer 2ce04d15fb77485293b97b7712700612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:24.232474 2640 consensus_queue.cc:1035] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:24.232611 2637 consensus_queue.cc:1035] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:24.235731 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333794792373719040 to 7333794791635582976. Current Snapshot: MvccSnapshot[applied={T|T < 7333794792373719040}]
I20260927 02:47:24.242542 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.240005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35759 (local address 127.0.98.1:36017)
0927 02:47:24.240044 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:24.240046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:24.240106 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:47:24.240173 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:24.240173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:24.240182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:24.240221 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:24.240346 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.240793 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.241881 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.242134 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.242197 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.242360 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:24.242387 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:24.242416 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:24.242473 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:47:24.242488 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:47:24.242589 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:24.239962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36017 (local address 127.0.98.3:35759)
0927 02:47:24.240064 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:24.240076 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:24.240120 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:24.240226 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:24.240227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:24.240228 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:24.240295 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.240296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.240863 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.240864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.241777 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:24.241779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:24.242202 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:24.242203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:24.242273 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:24.242289 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:24.242424 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:24.242425 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:47:24.242471 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:47:24.242513 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 02:47:24.251772 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:24.251892 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:24.252620 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Sending DeleteTablet for 3 replicas of tablet 741a769e0495470081e5e05f625ae184
I20260927 02:47:24.253017 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 741a769e0495470081e5e05f625ae184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46406
I20260927 02:47:24.253062 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 741a769e0495470081e5e05f625ae184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60160
I20260927 02:47:24.253261 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 741a769e0495470081e5e05f625ae184 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52740
I20260927 02:47:24.253497 2677 tablet_replica.cc:330] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: stopping tablet replica
I20260927 02:47:24.253611 2677 raft_consensus.cc:2229] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:24.253667 2677 raft_consensus.cc:2258] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:24.253957 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:24.254098 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:24.254103 2677 ts_tablet_manager.cc:1905] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:24.255550 2540 tablet_service.cc:1467] Processing CreateTablet for tablet c405c3eeded24ae1b85ecfe2fb985534 (DEFAULT_TABLE table=test-table [id=ea3b81acbc494d5a8432d424a3559997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:24.255668 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c405c3eeded24ae1b85ecfe2fb985534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:24.255764 2677 ts_tablet_manager.cc:1918] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:24.255823 2677 log.cc:1196] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/741a769e0495470081e5e05f625ae184
I20260927 02:47:24.256053 2677 ts_tablet_manager.cc:1939] T 741a769e0495470081e5e05f625ae184 P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting consensus metadata
I20260927 02:47:24.256584 2108 catalog_manager.cc:4841] TS 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401): tablet 741a769e0495470081e5e05f625ae184 (table test-table [id=a1a045f35e7a4a3a8774a269385aae0b]) successfully deleted
I20260927 02:47:24.256932 2629 tablet_bootstrap.cc:492] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: Bootstrap starting.
I20260927 02:47:24.257109 2400 tablet_service.cc:1467] Processing CreateTablet for tablet c405c3eeded24ae1b85ecfe2fb985534 (DEFAULT_TABLE table=test-table [id=ea3b81acbc494d5a8432d424a3559997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:24.257207 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c405c3eeded24ae1b85ecfe2fb985534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:24.257253 2629 tablet_bootstrap.cc:654] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:24.258337 2630 tablet_bootstrap.cc:492] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: Bootstrap starting.
I20260927 02:47:24.258358 2260 tablet_service.cc:1467] Processing CreateTablet for tablet c405c3eeded24ae1b85ecfe2fb985534 (DEFAULT_TABLE table=test-table [id=ea3b81acbc494d5a8432d424a3559997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:24.258461 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c405c3eeded24ae1b85ecfe2fb985534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:24.258675 2630 tablet_bootstrap.cc:654] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:24.259363 2630 tablet_bootstrap.cc:492] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: No bootstrap required, opened a new log
I20260927 02:47:24.259400 2630 ts_tablet_manager.cc:1397] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:24.259527 2630 raft_consensus.cc:348] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.259585 2630 raft_consensus.cc:374] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:24.259601 2630 raft_consensus.cc:729] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116a4a35293e43e4b3bd4d76c55d5f3f, State: Initialized, Role: FOLLOWER
I20260927 02:47:24.259641 2630 consensus_queue.cc:260] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.259696 2631 tablet_bootstrap.cc:492] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: Bootstrap starting.
I20260927 02:47:24.259711 2630 ts_tablet_manager.cc:1428] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:24.260072 2631 tablet_bootstrap.cc:654] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:24.260673 2631 tablet_bootstrap.cc:492] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: No bootstrap required, opened a new log
I20260927 02:47:24.260720 2631 ts_tablet_manager.cc:1397] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:24.260871 2631 raft_consensus.cc:348] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.260936 2631 raft_consensus.cc:374] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:24.260954 2631 raft_consensus.cc:729] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce04d15fb77485293b97b7712700612, State: Initialized, Role: FOLLOWER
I20260927 02:47:24.260989 2631 consensus_queue.cc:260] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.261073 2631 ts_tablet_manager.cc:1428] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:24.262061 2678 tablet_replica.cc:330] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: stopping tablet replica
I20260927 02:47:24.262205 2678 raft_consensus.cc:2229] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:24.262343 2678 raft_consensus.cc:2258] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:24.263096 2678 ts_tablet_manager.cc:1905] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:24.264756 2678 ts_tablet_manager.cc:1918] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:24.264827 2678 log.cc:1196] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/741a769e0495470081e5e05f625ae184
I20260927 02:47:24.265021 2676 tablet_replica.cc:330] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: stopping tablet replica
I20260927 02:47:24.265101 2678 ts_tablet_manager.cc:1939] T 741a769e0495470081e5e05f625ae184 P 2ce04d15fb77485293b97b7712700612: Deleting consensus metadata
I20260927 02:47:24.265201 2676 raft_consensus.cc:2229] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:24.265297 2676 raft_consensus.cc:2258] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:24.265491 2109 catalog_manager.cc:4841] TS 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017): tablet 741a769e0495470081e5e05f625ae184 (table test-table [id=a1a045f35e7a4a3a8774a269385aae0b]) successfully deleted
I20260927 02:47:24.265813 2676 ts_tablet_manager.cc:1905] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:24.265941 2629 tablet_bootstrap.cc:492] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: No bootstrap required, opened a new log
I20260927 02:47:24.266043 2629 ts_tablet_manager.cc:1397] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260927 02:47:24.266160 2629 raft_consensus.cc:348] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.266240 2629 raft_consensus.cc:374] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:24.266265 2629 raft_consensus.cc:729] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0705588e87d45a6978d68ecece2a877, State: Initialized, Role: FOLLOWER
I20260927 02:47:24.266305 2629 consensus_queue.cc:260] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.266394 2629 ts_tablet_manager.cc:1428] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:24.268206 2676 ts_tablet_manager.cc:1918] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:24.268316 2676 log.cc:1196] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/741a769e0495470081e5e05f625ae184
I20260927 02:47:24.268595 2676 ts_tablet_manager.cc:1939] T 741a769e0495470081e5e05f625ae184 P c0705588e87d45a6978d68ecece2a877: Deleting consensus metadata
I20260927 02:47:24.268697 2674 raft_consensus.cc:482] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:24.268754 2674 raft_consensus.cc:504] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.268899 2674 leader_election.cc:288] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401), c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409)
I20260927 02:47:24.269052 2110 catalog_manager.cc:4841] TS c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409): tablet 741a769e0495470081e5e05f625ae184 (table test-table [id=a1a045f35e7a4a3a8774a269385aae0b]) successfully deleted
I20260927 02:47:24.269068 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405c3eeded24ae1b85ecfe2fb985534" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877" is_pre_election: true
I20260927 02:47:24.269132 2560 raft_consensus.cc:2454] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 0.
I20260927 02:47:24.269114 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405c3eeded24ae1b85ecfe2fb985534" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" is_pre_election: true
I20260927 02:47:24.269186 2420 raft_consensus.cc:2454] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 0.
I20260927 02:47:24.269275 2214 leader_election.cc:302] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce04d15fb77485293b97b7712700612, c0705588e87d45a6978d68ecece2a877; no voters:
I20260927 02:47:24.269403 2674 raft_consensus.cc:2781] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:24.269443 2674 raft_consensus.cc:482] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:24.269462 2674 raft_consensus.cc:3037] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.270138 2674 raft_consensus.cc:504] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.270278 2674 leader_election.cc:288] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 election: Requested vote from peers 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401), c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409)
I20260927 02:47:24.270437 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405c3eeded24ae1b85ecfe2fb985534" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f"
I20260927 02:47:24.270507 2420 raft_consensus.cc:3037] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.270491 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405c3eeded24ae1b85ecfe2fb985534" candidate_uuid: "2ce04d15fb77485293b97b7712700612" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877"
I20260927 02:47:24.270545 2560 raft_consensus.cc:3037] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:24.271168 2560 raft_consensus.cc:2454] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 1.
I20260927 02:47:24.271170 2420 raft_consensus.cc:2454] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce04d15fb77485293b97b7712700612 in term 1.
I20260927 02:47:24.271333 2214 leader_election.cc:302] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce04d15fb77485293b97b7712700612, c0705588e87d45a6978d68ecece2a877; no voters:
I20260927 02:47:24.271435 2674 raft_consensus.cc:2781] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:24.271484 2674 raft_consensus.cc:686] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce04d15fb77485293b97b7712700612, State: Running, Role: LEADER
I20260927 02:47:24.271530 2674 consensus_queue.cc:237] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:24.272053 2121 catalog_manager.cc:5495] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ce04d15fb77485293b97b7712700612 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2ce04d15fb77485293b97b7712700612" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:24.277781 2560 raft_consensus.cc:1261] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Refusing update from remote peer 2ce04d15fb77485293b97b7712700612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:24.277948 2420 raft_consensus.cc:1261] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Refusing update from remote peer 2ce04d15fb77485293b97b7712700612: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:24.278007 2674 consensus_queue.cc:1035] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:24.278244 2674 consensus_queue.cc:1035] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [LEADER]: Connected to new peer: Peer: permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:47:24.282783 2658 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:47:25.801115 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:25.801226 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:25.802028 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Sending DeleteTablet for 3 replicas of tablet c405c3eeded24ae1b85ecfe2fb985534
I20260927 02:47:25.802277 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet c405c3eeded24ae1b85ecfe2fb985534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46406
I20260927 02:47:25.802327 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet c405c3eeded24ae1b85ecfe2fb985534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60160
I20260927 02:47:25.802521 2708 tablet_replica.cc:330] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: stopping tablet replica
I20260927 02:47:25.802577 2708 raft_consensus.cc:2229] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:25.802618 2708 raft_consensus.cc:2258] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:25.802874 2708 ts_tablet_manager.cc:1905] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:25.804006 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet c405c3eeded24ae1b85ecfe2fb985534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52740
I20260927 02:47:25.804198 2710 tablet_replica.cc:330] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: stopping tablet replica
I20260927 02:47:25.804288 2710 raft_consensus.cc:2229] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:25.804379 2710 raft_consensus.cc:2258] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:25.804415 2708 ts_tablet_manager.cc:1918] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:47:25.804476 2708 log.cc:1196] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/c405c3eeded24ae1b85ecfe2fb985534
I20260927 02:47:25.804692 2708 ts_tablet_manager.cc:1939] T c405c3eeded24ae1b85ecfe2fb985534 P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting consensus metadata
I20260927 02:47:25.804718 2710 ts_tablet_manager.cc:1905] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:25.804840 2707 tablet_replica.cc:330] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: stopping tablet replica
I20260927 02:47:25.804901 2707 raft_consensus.cc:2229] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:25.804941 2707 raft_consensus.cc:2258] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:25.805022 2108 catalog_manager.cc:4841] TS 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401): tablet c405c3eeded24ae1b85ecfe2fb985534 (table test-table [id=ea3b81acbc494d5a8432d424a3559997]) successfully deleted
I20260927 02:47:25.805152 2707 ts_tablet_manager.cc:1905] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:25.806475 2707 ts_tablet_manager.cc:1918] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:47:25.806497 2710 ts_tablet_manager.cc:1918] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:47:25.806527 2707 log.cc:1196] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/c405c3eeded24ae1b85ecfe2fb985534
I20260927 02:47:25.806542 2710 log.cc:1196] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/c405c3eeded24ae1b85ecfe2fb985534
I20260927 02:47:25.806715 2707 ts_tablet_manager.cc:1939] T c405c3eeded24ae1b85ecfe2fb985534 P c0705588e87d45a6978d68ecece2a877: Deleting consensus metadata
I20260927 02:47:25.806879 2710 ts_tablet_manager.cc:1939] T c405c3eeded24ae1b85ecfe2fb985534 P 2ce04d15fb77485293b97b7712700612: Deleting consensus metadata
I20260927 02:47:25.807262 2109 catalog_manager.cc:4841] TS 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017): tablet c405c3eeded24ae1b85ecfe2fb985534 (table test-table [id=ea3b81acbc494d5a8432d424a3559997]) successfully deleted
I20260927 02:47:25.807346 2110 catalog_manager.cc:4841] TS c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409): tablet c405c3eeded24ae1b85ecfe2fb985534 (table test-table [id=ea3b81acbc494d5a8432d424a3559997]) successfully deleted
Sep 27 02:47:25 dist-test-slave-f4rg krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:47:25 dist-test-slave-f4rg krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477245, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:25.823988 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:25.817570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.0.98.62:36677)
0927 02:47:25.817694 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:25.817696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:25.817705 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:25.817762 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:25.817763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:25.817775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:25.817849 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:25.818366 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:25.818932 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:25.819616 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:25.819806 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:25.819846 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:25.821832 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:25.821833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:25.821834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:25.821841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:25.823167 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:25.823452 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:25.823452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:25.823453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:25.823501 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:25.823628 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:25.823628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:25.823628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:25.823721 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:25.823790 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:25.823860 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:47:25.823919 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260927 02:47:25.825264 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:25.825409 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:25.826596 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 34d1dd19958a4854af459d194f36626e (DEFAULT_TABLE table=test-table [id=cb9874fbf0724b208952f2f937cef1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:25.826617 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 34d1dd19958a4854af459d194f36626e (DEFAULT_TABLE table=test-table [id=cb9874fbf0724b208952f2f937cef1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:25.826730 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d1dd19958a4854af459d194f36626e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:25.826730 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d1dd19958a4854af459d194f36626e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:25.826921 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 34d1dd19958a4854af459d194f36626e (DEFAULT_TABLE table=test-table [id=cb9874fbf0724b208952f2f937cef1e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:25.827035 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d1dd19958a4854af459d194f36626e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:25.828001 2718 tablet_bootstrap.cc:492] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: Bootstrap starting.
I20260927 02:47:25.828244 2719 tablet_bootstrap.cc:492] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: Bootstrap starting.
I20260927 02:47:25.828420 2718 tablet_bootstrap.cc:654] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:25.828604 2719 tablet_bootstrap.cc:654] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:25.828915 2720 tablet_bootstrap.cc:492] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: Bootstrap starting.
I20260927 02:47:25.828989 2718 tablet_bootstrap.cc:492] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: No bootstrap required, opened a new log
I20260927 02:47:25.829044 2718 ts_tablet_manager.cc:1397] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:25.829131 2719 tablet_bootstrap.cc:492] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: No bootstrap required, opened a new log
I20260927 02:47:25.829182 2719 ts_tablet_manager.cc:1397] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:25.829249 2720 tablet_bootstrap.cc:654] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:25.829206 2718 raft_consensus.cc:348] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:25.829282 2718 raft_consensus.cc:374] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:25.829306 2718 raft_consensus.cc:729] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116a4a35293e43e4b3bd4d76c55d5f3f, State: Initialized, Role: FOLLOWER
I20260927 02:47:25.829300 2719 raft_consensus.cc:348] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:25.829346 2718 consensus_queue.cc:260] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:25.829381 2719 raft_consensus.cc:374] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:25.829407 2719 raft_consensus.cc:729] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce04d15fb77485293b97b7712700612, State: Initialized, Role: FOLLOWER
I20260927 02:47:25.829469 2719 consensus_queue.cc:260] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:25.829510 2718 ts_tablet_manager.cc:1428] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:25.829698 2719 ts_tablet_manager.cc:1428] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:25.829761 2720 tablet_bootstrap.cc:492] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: No bootstrap required, opened a new log
I20260927 02:47:25.829804 2720 ts_tablet_manager.cc:1397] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:25.829918 2720 raft_consensus.cc:348] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:25.829986 2720 raft_consensus.cc:374] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:25.830005 2720 raft_consensus.cc:729] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0705588e87d45a6978d68ecece2a877, State: Initialized, Role: FOLLOWER
I20260927 02:47:25.830046 2720 consensus_queue.cc:260] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:25.830138 2720 ts_tablet_manager.cc:1428] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:26.146581 2724 raft_consensus.cc:482] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:26.146672 2724 raft_consensus.cc:504] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.146996 2724 leader_election.cc:288] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017), c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409)
I20260927 02:47:26.149937 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.147234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46525 (local address 127.0.98.1:36017)
0927 02:47:26.147433 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:26.147435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:26.147470 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:47:26.147553 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:26.147554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:26.147564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:26.147624 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:26.147777 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.148249 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.149240 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.149620 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.149677 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.149690 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:26.149824 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:47:26.149860 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":107,"threads_started":1}
I20260927 02:47:26.149950 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.147256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54221 (local address 127.0.98.3:40409)
0927 02:47:26.147443 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:26.147445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:26.147483 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:47:26.147553 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:26.147554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:26.147564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:26.147620 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:26.147787 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.148248 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.149242 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.149632 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.149691 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.149703 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:26.149839 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:47:26.149876 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260927 02:47:26.149973 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.147191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40409 (local address 127.0.98.2:54221)
0927 02:47:26.147407 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:26.147419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:26.147486 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:26.147633 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:26.147634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:26.147639 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:26.147726 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.147727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.148324 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.148325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.149160 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.149162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.149725 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.149726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.149779 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.149782 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:26.149827 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:47:26.149879 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":17,"thread_start_us":165,"threads_started":1}
I20260927 02:47:26.149984 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.147178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36017 (local address 127.0.98.2:46525)
0927 02:47:26.147405 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:26.147418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:26.147487 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:26.147633 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:26.147634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:26.147636 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:26.147721 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.147723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.148323 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.148324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.149163 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.149165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.149688 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.149689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.149767 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.149771 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:26.149814 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:47:26.149857 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":44,"thread_start_us":184,"threads_started":1}
I20260927 02:47:26.150223 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d1dd19958a4854af459d194f36626e" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877" is_pre_election: true
I20260927 02:47:26.150324 2560 raft_consensus.cc:2454] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 0.
I20260927 02:47:26.150365 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d1dd19958a4854af459d194f36626e" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce04d15fb77485293b97b7712700612" is_pre_election: true
I20260927 02:47:26.150446 2280 raft_consensus.cc:2454] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 0.
I20260927 02:47:26.150509 2354 leader_election.cc:302] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, c0705588e87d45a6978d68ecece2a877; no voters:
I20260927 02:47:26.150628 2724 raft_consensus.cc:2781] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:26.150666 2724 raft_consensus.cc:482] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:26.150686 2724 raft_consensus.cc:3037] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:26.151367 2724 raft_consensus.cc:504] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.151489 2724 leader_election.cc:288] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 election: Requested vote from peers 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017), c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409)
I20260927 02:47:26.151619 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d1dd19958a4854af459d194f36626e" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce04d15fb77485293b97b7712700612"
I20260927 02:47:26.151691 2280 raft_consensus.cc:3037] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:26.151671 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d1dd19958a4854af459d194f36626e" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877"
I20260927 02:47:26.151746 2560 raft_consensus.cc:3037] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:26.152249 2280 raft_consensus.cc:2454] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 1.
I20260927 02:47:26.152305 2560 raft_consensus.cc:2454] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 1.
I20260927 02:47:26.152429 2353 leader_election.cc:302] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, 2ce04d15fb77485293b97b7712700612; no voters:
I20260927 02:47:26.152529 2724 raft_consensus.cc:2781] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:26.152663 2724 raft_consensus.cc:686] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 LEADER]: Becoming Leader. State: Replica: 116a4a35293e43e4b3bd4d76c55d5f3f, State: Running, Role: LEADER
I20260927 02:47:26.152724 2724 consensus_queue.cc:237] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.153353 2123 catalog_manager.cc:5495] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:26.184052 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.181580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.0.98.2:38401)
0927 02:47:26.181724 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:26.181726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:26.181731 (+ 5us) server_negotiation.cc:407] Connection header received
0927 02:47:26.181763 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:26.181764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:26.181782 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:26.181856 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:26.181979 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.182477 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.183251 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.183495 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.183550 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.183699 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:26.183810 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:26.183875 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:26.183921 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:47:26.183941 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260927 02:47:26.188325 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.186270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48599 (local address 127.0.98.3:40409)
0927 02:47:26.186319 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:26.186321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:26.186413 (+ 92us) server_negotiation.cc:407] Connection header received
0927 02:47:26.186450 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:26.186451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:26.186462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:26.186502 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:26.186614 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.187091 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.187788 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.187973 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.188020 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.188150 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:26.188174 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:26.188208 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:26.188251 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 02:47:26.188267 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:47:26.188344 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.186204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40409 (local address 127.0.98.2:48599)
0927 02:47:26.186354 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:26.186363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:26.186412 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:26.186510 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:26.186511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:26.186512 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:26.186557 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.186558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.187154 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.187155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.187724 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.187725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.188029 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.188029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.188075 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.188089 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:26.188217 (+ 128us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:26.188219 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:47:26.188248 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 02:47:26.188278 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260927 02:47:26.190469 2280 raft_consensus.cc:1261] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Refusing update from remote peer 116a4a35293e43e4b3bd4d76c55d5f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:26.190665 2560 raft_consensus.cc:1261] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Refusing update from remote peer 116a4a35293e43e4b3bd4d76c55d5f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:26.190889 2729 consensus_queue.cc:1035] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:26.191018 2724 consensus_queue.cc:1035] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:26.197824 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.195567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38401 (local address 127.0.98.3:52573)
0927 02:47:26.195690 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:26.195701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:26.195756 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:26.195875 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:26.195876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:26.195877 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:47:26.195925 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.195927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.196506 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.196507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.197070 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:26.197072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.197462 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.197463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:26.197521 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.197525 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:47:26.197668 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:47:26.197670 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:47:26.197705 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 02:47:26.197752 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":82,"threads_started":1}
I20260927 02:47:26.197813 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:26.195616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52573 (local address 127.0.98.2:38401)
0927 02:47:26.195657 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:26.195659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:26.195748 (+ 89us) server_negotiation.cc:407] Connection header received
0927 02:47:26.195810 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:26.195810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:26.195821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:26.195872 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:26.195976 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.196439 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.197148 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:26.197400 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:26.197455 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:26.197585 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:26.197619 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:26.197659 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:26.197716 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:47:26.197736 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:47:26.204365 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:26.204458 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:26.205034 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Sending DeleteTablet for 3 replicas of tablet 34d1dd19958a4854af459d194f36626e
I20260927 02:47:26.205251 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d1dd19958a4854af459d194f36626e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52740
I20260927 02:47:26.205292 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d1dd19958a4854af459d194f36626e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46406
I20260927 02:47:26.205392 2710 tablet_replica.cc:330] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: stopping tablet replica
I20260927 02:47:26.205449 2710 raft_consensus.cc:2229] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:26.205480 2710 raft_consensus.cc:2258] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:26.205544 2707 tablet_replica.cc:330] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: stopping tablet replica
I20260927 02:47:26.205600 2707 raft_consensus.cc:2229] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:26.205637 2707 raft_consensus.cc:2258] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:26.205667 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:47:26.205772 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d1dd19958a4854af459d194f36626e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60160
W20260927 02:47:26.205818 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:26.205855 2708 tablet_replica.cc:330] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: stopping tablet replica
I20260927 02:47:26.205895 2708 raft_consensus.cc:2229] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:26.205968 2708 raft_consensus.cc:2258] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:26.206362 2710 ts_tablet_manager.cc:1905] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:26.206893 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 98de11bf3c044047876718ddb187996d (DEFAULT_TABLE table=test-table [id=f7757c41a05146ce83dfab47c490370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:26.207002 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98de11bf3c044047876718ddb187996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:26.207031 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 98de11bf3c044047876718ddb187996d (DEFAULT_TABLE table=test-table [id=f7757c41a05146ce83dfab47c490370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:26.207134 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98de11bf3c044047876718ddb187996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:26.207942 2718 tablet_bootstrap.cc:492] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: Bootstrap starting.
I20260927 02:47:26.208052 2707 ts_tablet_manager.cc:1905] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:26.208338 2718 tablet_bootstrap.cc:654] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:26.209180 2719 tablet_bootstrap.cc:492] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612: Bootstrap starting.
I20260927 02:47:26.209488 2719 tablet_bootstrap.cc:654] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:26.209734 2707 ts_tablet_manager.cc:1918] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:26.209784 2707 log.cc:1196] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/34d1dd19958a4854af459d194f36626e
I20260927 02:47:26.209841 2708 ts_tablet_manager.cc:1905] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:26.210006 2707 ts_tablet_manager.cc:1939] T 34d1dd19958a4854af459d194f36626e P c0705588e87d45a6978d68ecece2a877: Deleting consensus metadata
I20260927 02:47:26.210175 2718 tablet_bootstrap.cc:492] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: No bootstrap required, opened a new log
I20260927 02:47:26.210214 2718 ts_tablet_manager.cc:1397] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:26.210322 2110 catalog_manager.cc:4841] TS c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409): tablet 34d1dd19958a4854af459d194f36626e (table test-table [id=cb9874fbf0724b208952f2f937cef1e3]) successfully deleted
I20260927 02:47:26.210352 2718 raft_consensus.cc:348] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.210424 2718 raft_consensus.cc:374] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:26.210443 2718 raft_consensus.cc:729] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116a4a35293e43e4b3bd4d76c55d5f3f, State: Initialized, Role: FOLLOWER
I20260927 02:47:26.210482 2718 consensus_queue.cc:260] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.210577 2718 ts_tablet_manager.cc:1428] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:26.211159 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 98de11bf3c044047876718ddb187996d (DEFAULT_TABLE table=test-table [id=f7757c41a05146ce83dfab47c490370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:26.211213 2708 ts_tablet_manager.cc:1918] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:26.211261 2708 log.cc:1196] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/34d1dd19958a4854af459d194f36626e
I20260927 02:47:26.211269 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98de11bf3c044047876718ddb187996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:26.211462 2708 ts_tablet_manager.cc:1939] T 34d1dd19958a4854af459d194f36626e P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting consensus metadata
I20260927 02:47:26.211849 2108 catalog_manager.cc:4841] TS 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401): tablet 34d1dd19958a4854af459d194f36626e (table test-table [id=cb9874fbf0724b208952f2f937cef1e3]) successfully deleted
I20260927 02:47:26.212265 2720 tablet_bootstrap.cc:492] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: Bootstrap starting.
I20260927 02:47:26.212608 2720 tablet_bootstrap.cc:654] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:26.213145 2720 tablet_bootstrap.cc:492] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: No bootstrap required, opened a new log
I20260927 02:47:26.213198 2720 ts_tablet_manager.cc:1397] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:26.213310 2710 ts_tablet_manager.cc:1918] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:47:26.213292 2720 raft_consensus.cc:348] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.213361 2710 log.cc:1196] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/34d1dd19958a4854af459d194f36626e
I20260927 02:47:26.213362 2720 raft_consensus.cc:374] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:26.213384 2720 raft_consensus.cc:729] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0705588e87d45a6978d68ecece2a877, State: Initialized, Role: FOLLOWER
I20260927 02:47:26.213423 2720 consensus_queue.cc:260] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.213507 2720 ts_tablet_manager.cc:1428] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:26.213567 2710 ts_tablet_manager.cc:1939] T 34d1dd19958a4854af459d194f36626e P 2ce04d15fb77485293b97b7712700612: Deleting consensus metadata
I20260927 02:47:26.214011 2109 catalog_manager.cc:4841] TS 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017): tablet 34d1dd19958a4854af459d194f36626e (table test-table [id=cb9874fbf0724b208952f2f937cef1e3]) successfully deleted
I20260927 02:47:26.214036 2719 tablet_bootstrap.cc:492] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612: No bootstrap required, opened a new log
I20260927 02:47:26.214079 2719 ts_tablet_manager.cc:1397] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 02:47:26.214184 2719 raft_consensus.cc:348] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.214257 2719 raft_consensus.cc:374] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:26.214282 2719 raft_consensus.cc:729] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce04d15fb77485293b97b7712700612, State: Initialized, Role: FOLLOWER
I20260927 02:47:26.214329 2719 consensus_queue.cc:260] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.214418 2719 ts_tablet_manager.cc:1428] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:26.221731 2729 raft_consensus.cc:482] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:26.221815 2729 raft_consensus.cc:504] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.221978 2729 leader_election.cc:288] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017), c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409)
I20260927 02:47:26.222152 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de11bf3c044047876718ddb187996d" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877" is_pre_election: true
I20260927 02:47:26.222239 2560 raft_consensus.cc:2454] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 0.
I20260927 02:47:26.222196 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de11bf3c044047876718ddb187996d" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce04d15fb77485293b97b7712700612" is_pre_election: true
I20260927 02:47:26.222290 2280 raft_consensus.cc:2454] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 0.
I20260927 02:47:26.222379 2354 leader_election.cc:302] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, c0705588e87d45a6978d68ecece2a877; no voters:
I20260927 02:47:26.222467 2729 raft_consensus.cc:2781] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:26.222510 2729 raft_consensus.cc:482] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:26.222532 2729 raft_consensus.cc:3037] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:26.223315 2729 raft_consensus.cc:504] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.223433 2729 leader_election.cc:288] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 election: Requested vote from peers 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017), c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409)
I20260927 02:47:26.223577 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de11bf3c044047876718ddb187996d" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0705588e87d45a6978d68ecece2a877"
I20260927 02:47:26.223646 2560 raft_consensus.cc:3037] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:26.223634 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98de11bf3c044047876718ddb187996d" candidate_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ce04d15fb77485293b97b7712700612"
I20260927 02:47:26.223707 2280 raft_consensus.cc:3037] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:26.224238 2560 raft_consensus.cc:2454] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 1.
I20260927 02:47:26.224406 2354 leader_election.cc:302] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 116a4a35293e43e4b3bd4d76c55d5f3f, c0705588e87d45a6978d68ecece2a877; no voters:
I20260927 02:47:26.224479 2729 raft_consensus.cc:2781] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:26.224483 2280 raft_consensus.cc:2454] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 116a4a35293e43e4b3bd4d76c55d5f3f in term 1.
I20260927 02:47:26.224519 2729 raft_consensus.cc:686] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 LEADER]: Becoming Leader. State: Replica: 116a4a35293e43e4b3bd4d76c55d5f3f, State: Running, Role: LEADER
I20260927 02:47:26.224573 2729 consensus_queue.cc:237] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } }
I20260927 02:47:26.225135 2123 catalog_manager.cc:5495] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "116a4a35293e43e4b3bd4d76c55d5f3f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:26.231158 2560 raft_consensus.cc:1261] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Refusing update from remote peer 116a4a35293e43e4b3bd4d76c55d5f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:26.231287 2280 raft_consensus.cc:1261] T 98de11bf3c044047876718ddb187996d P 2ce04d15fb77485293b97b7712700612 [term 1 FOLLOWER]: Refusing update from remote peer 116a4a35293e43e4b3bd4d76c55d5f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:26.231447 2724 consensus_queue.cc:1035] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0705588e87d45a6978d68ecece2a877" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:26.231551 2729 consensus_queue.cc:1035] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ce04d15fb77485293b97b7712700612" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:47:26.236547 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 02:47:27.753039 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:27.753139 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:27.753832 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8de085e7e7ce4c938c630040d521eeaa: Sending DeleteTablet for 3 replicas of tablet 98de11bf3c044047876718ddb187996d
I20260927 02:47:27.754074 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 98de11bf3c044047876718ddb187996d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60160
I20260927 02:47:27.754078 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 98de11bf3c044047876718ddb187996d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46406
I20260927 02:47:27.754122 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 98de11bf3c044047876718ddb187996d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52740
I20260927 02:47:27.754298 2771 tablet_replica.cc:330] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: stopping tablet replica
I20260927 02:47:27.754354 2771 raft_consensus.cc:2229] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:27.754387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 2195
I20260927 02:47:27.754436 2771 raft_consensus.cc:2258] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:27.756649 2771 ts_tablet_manager.cc:1905] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:27.758224 2771 ts_tablet_manager.cc:1918] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:47:27.758279 2771 log.cc:1196] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/98de11bf3c044047876718ddb187996d
I20260927 02:47:27.758468 2771 ts_tablet_manager.cc:1939] T 98de11bf3c044047876718ddb187996d P 116a4a35293e43e4b3bd4d76c55d5f3f: Deleting consensus metadata
I20260927 02:47:27.758751 2108 catalog_manager.cc:4841] TS 116a4a35293e43e4b3bd4d76c55d5f3f (127.0.98.2:38401): tablet 98de11bf3c044047876718ddb187996d (table test-table [id=f7757c41a05146ce83dfab47c490370d]) successfully deleted
I20260927 02:47:27.758934 2769 tablet_replica.cc:330] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: stopping tablet replica
I20260927 02:47:27.758991 2769 raft_consensus.cc:2229] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:27.759035 2769 raft_consensus.cc:2258] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:27.759230 2769 ts_tablet_manager.cc:1905] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:47:27.759919 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36017 (error 108)
I20260927 02:47:27.760425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 2335
I20260927 02:47:27.760483 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:27.760257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36017 (local address 127.0.0.1:41062)
0927 02:47:27.760398 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:27.760446 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
W20260927 02:47:27.760553 2109 catalog_manager.cc:4584] TS 2ce04d15fb77485293b97b7712700612 (127.0.98.1:36017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98de11bf3c044047876718ddb187996d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36017: connect: Connection refused (error 111)
I20260927 02:47:27.761243 2769 ts_tablet_manager.cc:1918] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 02:47:27.761296 2769 log.cc:1196] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/98de11bf3c044047876718ddb187996d
I20260927 02:47:27.761474 2769 ts_tablet_manager.cc:1939] T 98de11bf3c044047876718ddb187996d P c0705588e87d45a6978d68ecece2a877: Deleting consensus metadata
I20260927 02:47:27.761943 2110 catalog_manager.cc:4841] TS c0705588e87d45a6978d68ecece2a877 (127.0.98.3:40409): tablet 98de11bf3c044047876718ddb187996d (table test-table [id=f7757c41a05146ce83dfab47c490370d]) successfully deleted
I20260927 02:47:27.766070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 2475
I20260927 02:47:27.770661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 2091
2026-09-27T02:47:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5719 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:27 dist-test-slave-f4rg krb5kdc[2776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:27 dist-test-slave-f4rg krb5kdc[2776](info): set up 2 sockets
Sep 27 02:47:27 dist-test-slave-f4rg krb5kdc[2776](info): commencing operation
krb5kdc: starting...
W20260927 02:47:28.718391 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:28 dist-test-slave-f4rg krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:28.825261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34291
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:28.902396 2789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:28.902607 2789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:28.902628 2789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:28.903815 2789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:28.903859 2789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:28.903872 2789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:28.903883 2789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:28.903893 2789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:28.905350 2789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:28.905544 2789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:28.905721 2789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:28.910183 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:28.910354 2789 server_base.cc:1034] running on GCE node
W20260927 02:47:28.910210 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:28.910192 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:28.910706 2789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:28.911015 2789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:28.912160 2789 hybrid_clock.cc:648] HybridClock initialized: now 1790477248912138 us; error 42 us; skew 500 ppm
Sep 27 02:47:28 dist-test-slave-f4rg krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:28.914561 2789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:28.914987 2789 webserver.cc:460] Webserver started at http://127.0.98.62:46485/ using document root <none> and password file <none>
I20260927 02:47:28.915162 2789 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:28.915201 2789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:28.915297 2789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:28.916290 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "cf624d76f1ef4db28e9686b4175d4b68"
format_stamp: "Formatted at 2026-09-27 02:47:28 on dist-test-slave-f4rg"
server_key: "9a1ea7e6f21188c4cae6b23ee50fc047"
server_key_iv: "5a10057c51bc1baa1c0991434731e06a"
server_key_version: "encryptionkey@0"
I20260927 02:47:28.916574 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf624d76f1ef4db28e9686b4175d4b68"
format_stamp: "Formatted at 2026-09-27 02:47:28 on dist-test-slave-f4rg"
server_key: "9a1ea7e6f21188c4cae6b23ee50fc047"
server_key_iv: "5a10057c51bc1baa1c0991434731e06a"
server_key_version: "encryptionkey@0"
I20260927 02:47:28.917716 2789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:28.918298 2804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:28.918432 2789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:28.918509 2789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "cf624d76f1ef4db28e9686b4175d4b68"
format_stamp: "Formatted at 2026-09-27 02:47:28 on dist-test-slave-f4rg"
server_key: "9a1ea7e6f21188c4cae6b23ee50fc047"
server_key_iv: "5a10057c51bc1baa1c0991434731e06a"
server_key_version: "encryptionkey@0"
I20260927 02:47:28.918561 2789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:28.930742 2789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:28.932607 2789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:28.932732 2789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:28.936216 2789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34291
I20260927 02:47:28.936254 2856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34291 every 8 connection(s)
I20260927 02:47:28.936712 2789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:28.937340 2857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:28.939445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 2789
I20260927 02:47:28.939538 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:28.939710 392 external_mini_cluster.cc:1428] Setting key b0348dccd83ba2eee0cc9814cf25ea6d
I20260927 02:47:28.939716 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68: Bootstrap starting.
I20260927 02:47:28.940330 2857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:28.940644 2857 log.cc:824] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:28.941320 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68: No bootstrap required, opened a new log
I20260927 02:47:28.943502 2857 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf624d76f1ef4db28e9686b4175d4b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34291 } }
I20260927 02:47:28.943676 2857 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:28.943708 2857 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf624d76f1ef4db28e9686b4175d4b68, State: Initialized, Role: FOLLOWER
I20260927 02:47:28.943825 2857 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf624d76f1ef4db28e9686b4175d4b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34291 } }
I20260927 02:47:28.943907 2857 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:28.943938 2857 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:28.943985 2857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:28.944555 2857 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf624d76f1ef4db28e9686b4175d4b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34291 } }
I20260927 02:47:28.944664 2857 leader_election.cc:302] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf624d76f1ef4db28e9686b4175d4b68; no voters:
I20260927 02:47:28.944823 2857 leader_election.cc:288] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:28.944980 2863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:28.945066 2857 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:28.945127 2863 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [term 1 LEADER]: Becoming Leader. State: Replica: cf624d76f1ef4db28e9686b4175d4b68, State: Running, Role: LEADER
I20260927 02:47:28.945240 2863 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf624d76f1ef4db28e9686b4175d4b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34291 } }
I20260927 02:47:28.945540 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf624d76f1ef4db28e9686b4175d4b68. Latest consensus state: current_term: 1 leader_uuid: "cf624d76f1ef4db28e9686b4175d4b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf624d76f1ef4db28e9686b4175d4b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34291 } } }
I20260927 02:47:28.945591 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf624d76f1ef4db28e9686b4175d4b68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf624d76f1ef4db28e9686b4175d4b68" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34291 } } }
I20260927 02:47:28.945627 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:28.945641 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68 [sys.catalog]: This master's current role is: LEADER
Sep 27 02:47:28 dist-test-slave-f4rg krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260927 02:47:28.946975 2878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:47:28.947031 2878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:47:28.947070 2868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:28.947211 2868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:28.948675 2868 catalog_manager.cc:1347] Generated new cluster ID: c5012cc79ba0478bbba73209c444f331
I20260927 02:47:28.948729 2868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:28.950294 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:28.940214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.0.98.62:34291)
0927 02:47:28.940372 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:28.940381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:28.940570 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:47:28.941035 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:28.941042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:28.941091 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:28.941212 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:28.941966 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:28.942848 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:28.943856 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:28.944341 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:28.944419 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:28.947082 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:28.947086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:28.947086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:28.947097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:28.948923 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:28.949232 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:28.949235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:28.949235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:28.949319 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:28.949483 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:28.949484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:28.949484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:28.949723 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:28.949866 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:28.949967 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:47:28.950091 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260927 02:47:28.974320 2868 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:28.975119 2868 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:28.981709 2868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf624d76f1ef4db28e9686b4175d4b68: Generated new TSK 0
I20260927 02:47:28.981927 2868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:29.013276 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34291
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:29.092571 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:29.092782 2883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:29.092813 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:29.093962 2883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:29.094002 2883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:29.094038 2883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:29.095593 2883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:29.095827 2883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:29.096026 2883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:29.100687 2891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:29.100682 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:29.100730 2883 server_base.cc:1034] running on GCE node
W20260927 02:47:29.100729 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:29.101110 2883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:29.101333 2883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:29.102473 2883 hybrid_clock.cc:648] HybridClock initialized: now 1790477249102450 us; error 36 us; skew 500 ppm
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:29.104830 2883 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:29.105245 2883 webserver.cc:460] Webserver started at http://127.0.98.1:34717/ using document root <none> and password file <none>
I20260927 02:47:29.105438 2883 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:29.105486 2883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:29.105608 2883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:29.106640 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "b585767c7a634d868ea22d73de24376e"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "504cf2219fcfcde5a96db392386188fa"
server_key_iv: "0e39542ffdfa6d22b3c7bfe060b470c0"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.107045 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b585767c7a634d868ea22d73de24376e"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "504cf2219fcfcde5a96db392386188fa"
server_key_iv: "0e39542ffdfa6d22b3c7bfe060b470c0"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.108220 2883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:29.108886 2898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.109023 2883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.109102 2883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "b585767c7a634d868ea22d73de24376e"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "504cf2219fcfcde5a96db392386188fa"
server_key_iv: "0e39542ffdfa6d22b3c7bfe060b470c0"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.109158 2883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:29.138671 2883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:29.140527 2883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:29.140655 2883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:29.140869 2883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:29.141181 2883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:29.141215 2883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.141244 2883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:29.141268 2883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.146858 2883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46615
I20260927 02:47:29.146925 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46615 every 8 connection(s)
I20260927 02:47:29.147231 2883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:29.148108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 2883
I20260927 02:47:29.148190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:29.148365 392 external_mini_cluster.cc:1428] Setting key 7a66d80bb5e5e7cf834799b8124ba2d0
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:47:29.156919 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:29.147948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44357 (local address 127.0.98.62:34291)
0927 02:47:29.148019 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:29.148021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:29.148219 (+ 198us) server_negotiation.cc:407] Connection header received
0927 02:47:29.149304 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:29.149305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:29.149317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:29.149384 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:29.150012 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.150520 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.151548 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.151775 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.151842 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:29.154142 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:29.154143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:29.154144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:29.154151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:29.155524 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:29.155911 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:29.155912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:29.155913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:29.155963 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:29.156179 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:29.156180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:29.156181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:29.156299 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:29.156354 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:29.156724 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 02:47:29.156820 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:47:29.157039 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:29.147923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34291 (local address 127.0.98.1:44357)
0927 02:47:29.148092 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:29.148137 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:29.149206 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:29.149444 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:29.149447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:29.149721 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:29.149929 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:29.149935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.150604 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.150605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:29.151470 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:29.151472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.151875 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.151877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:29.152042 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:29.152544 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:29.152554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:29.154051 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:29.155598 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:29.155605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:29.155607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:29.155847 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:29.156016 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:29.156016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:29.156017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:29.156127 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:29.156379 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:29.156386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:29.156550 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 02:47:29.156649 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 02:47:29.156823 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260927 02:47:29.157522 3012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34291
I20260927 02:47:29.157624 3012 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 02:47:29.157801 3012 heartbeater.cc:511] Master 127.0.98.62:34291 requested a full tablet report, sending...
I20260927 02:47:29.158326 2821 ts_manager.cc:194] Registered new tserver with Master: b585767c7a634d868ea22d73de24376e (127.0.98.1:46615)
I20260927 02:47:29.158968 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:29.171528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34291
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:29.250335 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:29.250550 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:29.250573 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:29.251703 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:29.251740 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:29.251765 3017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:29.253222 3017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:29.253413 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:29.253607 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:29.258215 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:29.258221 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:29.258260 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:29.258249 3017 server_base.cc:1034] running on GCE node
I20260927 02:47:29.258598 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:29.258872 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:29.260020 3017 hybrid_clock.cc:648] HybridClock initialized: now 1790477249259997 us; error 37 us; skew 500 ppm
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:29.262377 3017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:29.262773 3017 webserver.cc:460] Webserver started at http://127.0.98.2:33841/ using document root <none> and password file <none>
I20260927 02:47:29.263001 3017 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:29.263051 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:29.263176 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:29.264200 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "a94941802e534ffa9afe789729b5b759"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "59646eaf6e57ca64122e421c699cf75a"
server_key_iv: "a7e2cf8b95c9555c7eca9e0afded56a0"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.264518 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a94941802e534ffa9afe789729b5b759"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "59646eaf6e57ca64122e421c699cf75a"
server_key_iv: "a7e2cf8b95c9555c7eca9e0afded56a0"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.265720 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:29.266419 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.266568 3017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:29.266649 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "a94941802e534ffa9afe789729b5b759"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "59646eaf6e57ca64122e421c699cf75a"
server_key_iv: "a7e2cf8b95c9555c7eca9e0afded56a0"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.266716 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:29.296697 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:29.298645 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:29.298782 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:29.299031 3017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:29.299342 3017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:29.299381 3017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.299410 3017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:29.299434 3017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.304705 3017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42491
I20260927 02:47:29.304766 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42491 every 8 connection(s)
I20260927 02:47:29.305050 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:29.306231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3017
I20260927 02:47:29.306327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:29.306491 392 external_mini_cluster.cc:1428] Setting key 734e4485447de04e3804683643b6dd70
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:47:29.314670 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:29.305813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46863 (local address 127.0.98.62:34291)
0927 02:47:29.305888 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:29.305890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:29.306225 (+ 335us) server_negotiation.cc:407] Connection header received
0927 02:47:29.307335 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:29.307336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:29.307348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:29.307392 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:29.308008 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.308486 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.309445 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.309650 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.309705 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:29.312064 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:29.312065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:29.312066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:29.312072 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:29.313378 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:29.313843 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:29.313843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:29.313843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:29.313891 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:29.314143 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:29.314143 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:29.314144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:29.314251 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:29.314299 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:29.314543 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 02:47:29.314598 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 02:47:29.314880 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:29.305796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34291 (local address 127.0.98.2:46863)
0927 02:47:29.306065 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:29.306129 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:29.307245 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:29.307475 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:29.307479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:29.307740 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:29.307926 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:29.307930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.308564 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.308565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:29.309369 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:29.309372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.309718 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.309719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:29.309847 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:29.310345 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:29.310352 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:29.311975 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:29.313459 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:29.313466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:29.313468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:29.313776 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:29.313945 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:29.313946 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:29.313946 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:29.314080 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:29.314311 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:29.314316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:29.314442 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:47:29.314528 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:47:29.314645 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":129,"threads_started":1}
I20260927 02:47:29.315308 3146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34291
I20260927 02:47:29.315402 3146 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:29.315574 3146 heartbeater.cc:511] Master 127.0.98.62:34291 requested a full tablet report, sending...
I20260927 02:47:29.315899 2821 ts_manager.cc:194] Registered new tserver with Master: a94941802e534ffa9afe789729b5b759 (127.0.98.2:42491)
I20260927 02:47:29.316265 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:29.329319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34291
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:29.407667 3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:29.407869 3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:29.407900 3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:29.409044 3151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:29.409082 3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:29.409116 3151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:29.410585 3151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:29.410840 3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:29.411051 3151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:29.415668 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:29.415673 3157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:29.415774 3151 server_base.cc:1034] running on GCE node
W20260927 02:47:29.415742 3159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:29.416100 3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:29.416303 3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:29.417433 3151 hybrid_clock.cc:648] HybridClock initialized: now 1790477249417421 us; error 28 us; skew 500 ppm
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:29.419729 3151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:29.420123 3151 webserver.cc:460] Webserver started at http://127.0.98.3:34427/ using document root <none> and password file <none>
I20260927 02:47:29.420322 3151 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:29.420367 3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:29.420486 3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:29.421589 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "8565658c4e6344728b0f23042b7642f8"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "801aefcc1dc367dd6da2f75ca4558dc3"
server_key_iv: "e76746a5115c9148fd1f7eaef88a7ddf"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.421927 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8565658c4e6344728b0f23042b7642f8"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "801aefcc1dc367dd6da2f75ca4558dc3"
server_key_iv: "e76746a5115c9148fd1f7eaef88a7ddf"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.423177 3151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:29.423811 3166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.423954 3151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:29.424031 3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "8565658c4e6344728b0f23042b7642f8"
format_stamp: "Formatted at 2026-09-27 02:47:29 on dist-test-slave-f4rg"
server_key: "801aefcc1dc367dd6da2f75ca4558dc3"
server_key_iv: "e76746a5115c9148fd1f7eaef88a7ddf"
server_key_version: "encryptionkey@0"
I20260927 02:47:29.424088 3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:29.453110 3151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:29.455272 3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:29.455406 3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:29.455628 3151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:29.455909 3151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:29.455942 3151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.455971 3151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:29.455994 3151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:29.461617 3151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39003
I20260927 02:47:29.461669 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39003 every 8 connection(s)
I20260927 02:47:29.461977 3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:29.463551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3151
I20260927 02:47:29.463637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:29.463798 392 external_mini_cluster.cc:1428] Setting key aa30c5e637e94df74788dd768e7fa7e9
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:29.471208 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:29.462681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38733 (local address 127.0.98.62:34291)
0927 02:47:29.462738 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:29.462740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:29.462955 (+ 215us) server_negotiation.cc:407] Connection header received
0927 02:47:29.463893 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:29.463894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:29.463906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:29.463957 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:29.464670 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.465166 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.466139 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.466340 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.466395 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:29.468574 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:29.468576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:29.468576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:29.468583 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:29.469912 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:29.470315 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:29.470316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:29.470316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:29.470373 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:29.470578 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:29.470579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:29.470579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:29.470697 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:29.470749 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:29.471040 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 02:47:29.471116 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:47:29.471390 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:29.462655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34291 (local address 127.0.98.3:38733)
0927 02:47:29.462800 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:29.462866 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:29.463790 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:29.464037 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:29.464040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:29.464321 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:29.464583 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:29.464589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.465248 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.465249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:29.466056 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:29.466058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:29.466410 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:29.466411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:29.466531 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:29.467069 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:29.467078 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:29.468480 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:29.469985 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:29.469992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:29.469994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:29.470213 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:29.470423 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:29.470423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:29.470424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:29.470526 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:29.470747 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:29.470754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:29.470926 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 02:47:29.471014 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 02:47:29.471174 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260927 02:47:29.471859 3280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34291
I20260927 02:47:29.471942 3280 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:29.472112 3280 heartbeater.cc:511] Master 127.0.98.62:34291 requested a full tablet report, sending...
I20260927 02:47:29.472452 2821 ts_manager.cc:194] Registered new tserver with Master: 8565658c4e6344728b0f23042b7642f8 (127.0.98.3:39003)
I20260927 02:47:29.472786 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38733
I20260927 02:47:29.475345 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:47:29.485301 2860 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:29.484502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.0.98.62:34291)
0927 02:47:29.484549 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:29.484551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:29.484668 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:47:29.484717 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:29.484718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:29.484726 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:29.484780 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:29.485233 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53950: BlockingRecv error: recv got EOF from 127.0.0.1:53950 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:47:29.486234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 2883
I20260927 02:47:29.489902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3017
I20260927 02:47:29.493640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3151
I20260927 02:47:29.497160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 2789
2026-09-27T02:47:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1725 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[3292](info): set up 2 sockets
Sep 27 02:47:29 dist-test-slave-f4rg krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260927 02:47:30.438614 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:30 dist-test-slave-f4rg krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:30.550695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:30.628846 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:30.629063 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:30.629093 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:30.630306 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:30.630348 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:30.630366 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:30.630383 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:30.630398 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:30.631928 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:30.632167 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:30.632377 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:30.637126 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:30.637190 3305 server_base.cc:1034] running on GCE node
W20260927 02:47:30.637174 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:30.637207 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:30.637529 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:30.637802 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:30.638960 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790477250638946 us; error 33 us; skew 500 ppm
Sep 27 02:47:30 dist-test-slave-f4rg krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:30.641294 3305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:30.641692 3305 webserver.cc:460] Webserver started at http://127.0.98.62:40729/ using document root <none> and password file <none>
I20260927 02:47:30.641878 3305 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:30.641920 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:30.642004 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:30.643044 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "83acb07b5d1041438a7fcafc42322e4a"
format_stamp: "Formatted at 2026-09-27 02:47:30 on dist-test-slave-f4rg"
server_key: "a4e215719f1bf2dc90b9215fdb9a8d50"
server_key_iv: "92862b2332dcf997643b709e10f8b231"
server_key_version: "encryptionkey@0"
I20260927 02:47:30.643340 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "83acb07b5d1041438a7fcafc42322e4a"
format_stamp: "Formatted at 2026-09-27 02:47:30 on dist-test-slave-f4rg"
server_key: "a4e215719f1bf2dc90b9215fdb9a8d50"
server_key_iv: "92862b2332dcf997643b709e10f8b231"
server_key_version: "encryptionkey@0"
I20260927 02:47:30.644492 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:30.645135 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:30.645326 3305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:30.645411 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "83acb07b5d1041438a7fcafc42322e4a"
format_stamp: "Formatted at 2026-09-27 02:47:30 on dist-test-slave-f4rg"
server_key: "a4e215719f1bf2dc90b9215fdb9a8d50"
server_key_iv: "92862b2332dcf997643b709e10f8b231"
server_key_version: "encryptionkey@0"
I20260927 02:47:30.645479 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:30.675709 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:30.677497 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:30.677663 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:30.681341 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39379
I20260927 02:47:30.681380 3372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39379 every 8 connection(s)
I20260927 02:47:30.681694 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:30.682483 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:30.684772 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a: Bootstrap starting.
I20260927 02:47:30.685280 3373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:30.685524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3305
I20260927 02:47:30.685591 3373 log.cc:824] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:30.685618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:30.685781 392 external_mini_cluster.cc:1428] Setting key 8ec83f5bb531d8f6ba930b75f1b0a77a
I20260927 02:47:30.686316 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a: No bootstrap required, opened a new log
I20260927 02:47:30.688679 3373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83acb07b5d1041438a7fcafc42322e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260927 02:47:30.688851 3373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:30.688886 3373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83acb07b5d1041438a7fcafc42322e4a, State: Initialized, Role: FOLLOWER
I20260927 02:47:30.689008 3373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83acb07b5d1041438a7fcafc42322e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260927 02:47:30.689083 3373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:30.689116 3373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:30.689159 3373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:30.689798 3373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83acb07b5d1041438a7fcafc42322e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260927 02:47:30.689881 3373 leader_election.cc:302] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83acb07b5d1041438a7fcafc42322e4a; no voters:
I20260927 02:47:30.690023 3373 leader_election.cc:288] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:30.690059 3378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:30.690198 3378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [term 1 LEADER]: Becoming Leader. State: Replica: 83acb07b5d1041438a7fcafc42322e4a, State: Running, Role: LEADER
I20260927 02:47:30.690274 3373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:30.690325 3378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83acb07b5d1041438a7fcafc42322e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } }
I20260927 02:47:30.690610 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83acb07b5d1041438a7fcafc42322e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83acb07b5d1041438a7fcafc42322e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } } }
I20260927 02:47:30.690688 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:30.690828 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83acb07b5d1041438a7fcafc42322e4a. Latest consensus state: current_term: 1 leader_uuid: "83acb07b5d1041438a7fcafc42322e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83acb07b5d1041438a7fcafc42322e4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39379 } } }
I20260927 02:47:30.690954 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:30.690891 3384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:30.691615 3384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:30.693142 3384 catalog_manager.cc:1347] Generated new cluster ID: 3da9d76eddb34b19a9cb726396ccfead
I20260927 02:47:30.693190 3384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:47:30 dist-test-slave-f4rg krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:30.697487 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:30.686258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.0.98.62:39379)
0927 02:47:30.686417 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:30.686426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:30.686454 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:47:30.686903 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:30.686909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:30.686957 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:30.687066 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:30.687792 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:30.688645 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:30.689666 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:30.690155 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:30.690254 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:30.694031 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:30.694036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:30.694036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:30.694049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:30.695973 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:30.696337 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:30.696340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:30.696341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:30.696446 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:30.696620 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:30.696620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:30.696621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:30.696923 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:30.697081 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:30.697155 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:47:30.697292 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260927 02:47:30.698359 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:47:30.702514 3384 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:30.703063 3384 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:30.709205 3384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83acb07b5d1041438a7fcafc42322e4a: Generated new TSK 0
I20260927 02:47:30.709383 3384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:47:30.713627 3377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:30.709993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.98.62:39379)
0927 02:47:30.710038 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:30.710041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:30.710150 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:47:30.710219 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:30.710219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:30.710228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:30.710282 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:30.710707 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:30.711297 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:30.712362 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:30.712580 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:30.712632 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:30.713236 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:30.713237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:30.713237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:30.713409 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:30.713426 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:30.713431 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:30.713473 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:30.713547 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:47:30.713572 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:47:30.716029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3305
2026-09-27T02:47:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1217 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:30 dist-test-slave-f4rg krb5kdc[3406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:30 dist-test-slave-f4rg krb5kdc[3406](info): set up 2 sockets
Sep 27 02:47:30 dist-test-slave-f4rg krb5kdc[3406](info): commencing operation
krb5kdc: starting...
W20260927 02:47:31.658432 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:31 dist-test-slave-f4rg krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:31.767838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32941
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:31.846114 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:31.846336 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:31.846366 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:31.847572 3419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:31.847621 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:31.847640 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:31.847656 3419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:31.847669 3419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:31.849148 3419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:31.849388 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:31.849689 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:31.854257 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:31.854270 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:31.854326 3419 server_base.cc:1034] running on GCE node
W20260927 02:47:31.854283 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:31.854764 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:31.855041 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:31.856178 3419 hybrid_clock.cc:648] HybridClock initialized: now 1790477251856167 us; error 29 us; skew 500 ppm
Sep 27 02:47:31 dist-test-slave-f4rg krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:31.858511 3419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:31.858932 3419 webserver.cc:460] Webserver started at http://127.0.98.62:36793/ using document root <none> and password file <none>
I20260927 02:47:31.859113 3419 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:31.859151 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:31.859236 3419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:31.860229 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "187667293df940d3910a888d59ec1827"
format_stamp: "Formatted at 2026-09-27 02:47:31 on dist-test-slave-f4rg"
server_key: "1a94775e132b28e44df7e839405eee58"
server_key_iv: "fcd6cee47b06d8eb58208c25819ffc0d"
server_key_version: "encryptionkey@0"
I20260927 02:47:31.860512 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "187667293df940d3910a888d59ec1827"
format_stamp: "Formatted at 2026-09-27 02:47:31 on dist-test-slave-f4rg"
server_key: "1a94775e132b28e44df7e839405eee58"
server_key_iv: "fcd6cee47b06d8eb58208c25819ffc0d"
server_key_version: "encryptionkey@0"
I20260927 02:47:31.861647 3419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:31.862273 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:31.862390 3419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:31.862465 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "187667293df940d3910a888d59ec1827"
format_stamp: "Formatted at 2026-09-27 02:47:31 on dist-test-slave-f4rg"
server_key: "1a94775e132b28e44df7e839405eee58"
server_key_iv: "fcd6cee47b06d8eb58208c25819ffc0d"
server_key_version: "encryptionkey@0"
I20260927 02:47:31.862528 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:31.873975 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:31.876133 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:31.876266 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:31.879869 3419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32941
I20260927 02:47:31.879906 3486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32941 every 8 connection(s)
I20260927 02:47:31.880216 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:31.881136 3487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:31.882113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3419
I20260927 02:47:31.882239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:31.882419 392 external_mini_cluster.cc:1428] Setting key 30be5d74390102ce67ddc2136a74c472
I20260927 02:47:31.883713 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827: Bootstrap starting.
I20260927 02:47:31.884405 3487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:31.884685 3487 log.cc:824] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:31.885272 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827: No bootstrap required, opened a new log
I20260927 02:47:31.887456 3487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187667293df940d3910a888d59ec1827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260927 02:47:31.887632 3487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:31.887687 3487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187667293df940d3910a888d59ec1827, State: Initialized, Role: FOLLOWER
I20260927 02:47:31.887825 3487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187667293df940d3910a888d59ec1827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260927 02:47:31.887928 3487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:31.887967 3487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:31.888015 3487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:31.888623 3487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187667293df940d3910a888d59ec1827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260927 02:47:31.888722 3487 leader_election.cc:302] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 187667293df940d3910a888d59ec1827; no voters:
I20260927 02:47:31.888880 3487 leader_election.cc:288] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 27 02:47:31 dist-test-slave-f4rg krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:31.888921 3492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:31.889097 3492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [term 1 LEADER]: Becoming Leader. State: Replica: 187667293df940d3910a888d59ec1827, State: Running, Role: LEADER
I20260927 02:47:31.889112 3487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:31.889194 3492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187667293df940d3910a888d59ec1827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } }
I20260927 02:47:31.889566 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "187667293df940d3910a888d59ec1827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187667293df940d3910a888d59ec1827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } } }
I20260927 02:47:31.889566 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 187667293df940d3910a888d59ec1827. Latest consensus state: current_term: 1 leader_uuid: "187667293df940d3910a888d59ec1827" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "187667293df940d3910a888d59ec1827" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32941 } } }
I20260927 02:47:31.889717 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:31.889930 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:31.889724 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:31.890426 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:31.892514 3501 catalog_manager.cc:1347] Generated new cluster ID: 04e4f35226244504a288bc616415c785
I20260927 02:47:31.892562 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:31.892863 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:31.882958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.62:32941)
0927 02:47:31.883111 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:31.883121 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:31.883146 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:47:31.883556 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:31.883562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:31.883607 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:31.883725 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:31.884454 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:31.885409 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:31.886398 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:31.886776 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:31.886881 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:31.889463 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:31.889467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:31.889468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:31.889480 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:31.891336 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:31.891675 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:31.891678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:31.891678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:31.891789 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:31.892004 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:31.892005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:31.892005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:31.892322 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:31.892459 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:31.892585 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:47:31.892716 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260927 02:47:31.904927 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:31.905642 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:31.920387 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 187667293df940d3910a888d59ec1827: Generated new TSK 0
I20260927 02:47:31.920619 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:31.953505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:32.034276 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:32.034500 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:32.034533 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:32.035857 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:32.035921 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:32.035966 3513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:32.037563 3513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:32.037801 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:32.038017 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:32.042706 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:32.042724 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:32.042711 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:32.042922 3513 server_base.cc:1034] running on GCE node
I20260927 02:47:32.043130 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:32.043347 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:32.044484 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790477252044473 us; error 28 us; skew 500 ppm
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:32.047087 3513 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:32.047508 3513 webserver.cc:460] Webserver started at http://127.0.98.1:34111/ using document root <none> and password file <none>
I20260927 02:47:32.047703 3513 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:32.047745 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:32.047863 3513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:32.048900 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1b87dbf7858426eb38181cf38366fbf"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "f5c1a65907713908b77e730415b59451"
server_key_iv: "deb7b31300408cc0b6dd11c60f76292e"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.049242 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1b87dbf7858426eb38181cf38366fbf"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "f5c1a65907713908b77e730415b59451"
server_key_iv: "deb7b31300408cc0b6dd11c60f76292e"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.050441 3513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:47:32.051170 3528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.051321 3513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.051403 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "e1b87dbf7858426eb38181cf38366fbf"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "f5c1a65907713908b77e730415b59451"
server_key_iv: "deb7b31300408cc0b6dd11c60f76292e"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.051461 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:32.066391 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:32.068596 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:32.068735 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:32.068943 3513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:32.069249 3513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:32.069284 3513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.069314 3513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:32.069337 3513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.074965 3513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39775
I20260927 02:47:32.075017 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39775 every 8 connection(s)
I20260927 02:47:32.075325 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:32.078325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3513
I20260927 02:47:32.078418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:32.078583 392 external_mini_cluster.cc:1428] Setting key dfeb8c732d5b13229d54592e3f9fbe7b
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:32.084990 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:32.076005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48155 (local address 127.0.98.62:32941)
0927 02:47:32.076052 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:32.076054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:32.076263 (+ 209us) server_negotiation.cc:407] Connection header received
0927 02:47:32.076756 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:32.076757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:32.076767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:32.076828 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:32.077753 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.078254 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.079548 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.079800 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.079858 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:32.082026 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:32.082028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:32.082028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:32.082036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:32.083567 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.084037 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.084037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.084038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.084088 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.084324 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.084325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.084325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.084423 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:32.084466 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:32.084794 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 02:47:32.084919 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:47:32.085196 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:32.075982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32941 (local address 127.0.98.1:48155)
0927 02:47:32.076143 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:32.076185 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:32.076661 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:32.076898 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:32.076902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:32.077463 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:32.077658 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:32.077663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.078348 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.078349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:32.079468 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:32.079471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.079874 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.079875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:32.080002 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:32.080488 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:32.080496 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:32.081931 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:32.083640 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:32.083649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:32.083651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:32.083955 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:32.084126 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:32.084127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:32.084127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:32.084265 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:32.084475 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:32.084481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:32.084663 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 02:47:32.084784 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 02:47:32.084959 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260927 02:47:32.085772 3642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32941
I20260927 02:47:32.085865 3642 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:32.086035 3642 heartbeater.cc:511] Master 127.0.98.62:32941 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:47:32.086549 3451 ts_manager.cc:194] Registered new tserver with Master: e1b87dbf7858426eb38181cf38366fbf (127.0.98.1:39775)
I20260927 02:47:32.087222 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:32.101653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:32.181661 3647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:32.181859 3647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:32.181888 3647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:32.183070 3647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:32.183113 3647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:32.183148 3647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:32.184588 3647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:32.184834 3647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:32.185034 3647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:32.189813 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:32.189818 3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:32.189818 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:32.189977 3647 server_base.cc:1034] running on GCE node
I20260927 02:47:32.190227 3647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:32.190450 3647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:32.191599 3647 hybrid_clock.cc:648] HybridClock initialized: now 1790477252191576 us; error 37 us; skew 500 ppm
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:32.193862 3647 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:32.194247 3647 webserver.cc:460] Webserver started at http://127.0.98.2:43077/ using document root <none> and password file <none>
I20260927 02:47:32.194432 3647 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:32.194469 3647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:32.194558 3647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:32.195608 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "377ea007269d41dfa4a3629dd5a8eec2"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "481aee4851de8ea83f175301c786db9d"
server_key_iv: "f436c934a770d3160fb7c99d89d0e2fa"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.195886 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "377ea007269d41dfa4a3629dd5a8eec2"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "481aee4851de8ea83f175301c786db9d"
server_key_iv: "f436c934a770d3160fb7c99d89d0e2fa"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.197019 3647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:32.197642 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.197786 3647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:32.197865 3647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "377ea007269d41dfa4a3629dd5a8eec2"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "481aee4851de8ea83f175301c786db9d"
server_key_iv: "f436c934a770d3160fb7c99d89d0e2fa"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.197927 3647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:32.218663 3647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:32.220471 3647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:32.220607 3647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:32.220813 3647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:32.221107 3647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:32.221139 3647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.221169 3647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:32.221191 3647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.226688 3647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33123
I20260927 02:47:32.226752 3775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33123 every 8 connection(s)
I20260927 02:47:32.227105 3647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:32.236259 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3647
I20260927 02:47:32.236299 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:32.227844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47865 (local address 127.0.98.62:32941)
0927 02:47:32.227890 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:32.227892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:32.228099 (+ 207us) server_negotiation.cc:407] Connection header received
0927 02:47:32.228733 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:32.228735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:32.228747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:32.228806 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:32.229826 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.230336 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.231379 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.231584 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.231641 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:32.233818 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:32.233819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:32.233820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:32.233827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:32.234935 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.235400 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.235401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.235402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.235461 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.235694 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.235696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.235696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.235822 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:32.235894 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:32.236142 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:47:32.236216 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:47:32.236379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:32.236542 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:32.227811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32941 (local address 127.0.98.2:47865)
0927 02:47:32.227950 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:32.227990 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:32.228632 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:32.228880 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:32.228885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:32.229491 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:32.229693 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:32.229699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.230428 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.230429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:32.231294 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:32.231296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.231657 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.231658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:32.231789 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:32.232274 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:32.232282 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:32.233661 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:32.235003 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:32.235010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:32.235012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:32.235237 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:32.235519 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:32.235520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:32.235520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:32.235620 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:32.235889 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:32.235894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:32.236021 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 02:47:32.236128 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:47:32.236307 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260927 02:47:32.236570 392 external_mini_cluster.cc:1428] Setting key 6230c4627bf4a482153d792bedacf1b7
I20260927 02:47:32.237028 3776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32941
I20260927 02:47:32.237123 3776 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:32.237322 3776 heartbeater.cc:511] Master 127.0.98.62:32941 requested a full tablet report, sending...
I20260927 02:47:32.237686 3450 ts_manager.cc:194] Registered new tserver with Master: 377ea007269d41dfa4a3629dd5a8eec2 (127.0.98.2:33123)
I20260927 02:47:32.238044 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47865
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:32.259290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:32.337355 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:32.337567 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:32.337598 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:32.338752 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:32.338794 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:32.338862 3781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:32.340306 3781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:32.340539 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:32.340747 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:47:32.345463 3781 server_base.cc:1034] running on GCE node
W20260927 02:47:32.345438 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:32.345433 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:32.345443 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:32.345916 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:32.346138 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:32.347266 3781 hybrid_clock.cc:648] HybridClock initialized: now 1790477252347258 us; error 28 us; skew 500 ppm
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:32.349958 3781 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:32.350394 3781 webserver.cc:460] Webserver started at http://127.0.98.3:34579/ using document root <none> and password file <none>
I20260927 02:47:32.350590 3781 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:32.350637 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:32.350775 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:32.351882 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "c47c58390c1846028afe7d462a34b4ca"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "3098f0858b8f4646fd2aa9e77258641f"
server_key_iv: "407628c0dd1b82d28ee50e702e61f2ad"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.352212 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c47c58390c1846028afe7d462a34b4ca"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "3098f0858b8f4646fd2aa9e77258641f"
server_key_iv: "407628c0dd1b82d28ee50e702e61f2ad"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.353467 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:32.354130 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.354272 3781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:32.354354 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "c47c58390c1846028afe7d462a34b4ca"
format_stamp: "Formatted at 2026-09-27 02:47:32 on dist-test-slave-f4rg"
server_key: "3098f0858b8f4646fd2aa9e77258641f"
server_key_iv: "407628c0dd1b82d28ee50e702e61f2ad"
server_key_version: "encryptionkey@0"
I20260927 02:47:32.354422 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:32.372700 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:32.374898 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:32.375032 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:32.375247 3781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:32.375547 3781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:32.375578 3781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.375612 3781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:32.375638 3781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:32.380844 3781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46567
I20260927 02:47:32.380888 3909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46567 every 8 connection(s)
I20260927 02:47:32.381170 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:32.383628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3781
I20260927 02:47:32.383711 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:32.383877 392 external_mini_cluster.cc:1428] Setting key 1ab2daafa1a56c6cd70083cd58724e35
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:32.389876 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:32.381823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52461 (local address 127.0.98.62:32941)
0927 02:47:32.381882 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:32.381884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:32.382049 (+ 165us) server_negotiation.cc:407] Connection header received
0927 02:47:32.382520 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:32.382521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:32.382530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:32.382580 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:32.383507 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.384087 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.385274 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.385462 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.385513 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:32.387417 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:32.387419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:32.387419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:32.387426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:32.388647 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.389076 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.389077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.389077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.389125 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.389312 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.389313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.389313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.389407 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:32.389462 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:32.389756 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 02:47:32.389813 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:47:32.390110 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:32.381795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32941 (local address 127.0.98.3:52461)
0927 02:47:32.381943 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:32.381984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:32.382446 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:32.382642 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:32.382645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:32.383235 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:32.383428 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:32.383433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.384168 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.384169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:32.385202 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:32.385205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.385519 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.385520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:32.385618 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:32.386048 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:32.386056 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:32.387339 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:32.388730 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:32.388736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:32.388738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:32.389008 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:32.389171 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:32.389172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:32.389172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:32.389269 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:32.389520 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:32.389526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:32.389647 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 02:47:32.389742 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:47:32.389860 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260927 02:47:32.390523 3910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32941
I20260927 02:47:32.390604 3910 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:32.390744 3910 heartbeater.cc:511] Master 127.0.98.62:32941 requested a full tablet report, sending...
I20260927 02:47:32.391111 3450 ts_manager.cc:194] Registered new tserver with Master: c47c58390c1846028afe7d462a34b4ca (127.0.98.3:46567)
I20260927 02:47:32.391466 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52461
I20260927 02:47:32.395378 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:32.415963 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:32.409477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.0.98.62:32941)
0927 02:47:32.409536 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:32.409538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:32.409644 (+ 106us) server_negotiation.cc:407] Connection header received
0927 02:47:32.409698 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:32.409699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:32.409708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:32.409760 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:32.410253 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.410743 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.411495 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:32.411686 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:32.411740 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:32.413744 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:32.413745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:32.413746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:32.413752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:32.415060 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.415349 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.415350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.415350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.415406 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:32.415550 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:32.415550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:32.415551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:32.415690 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:32.415766 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:32.415838 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:47:32.415899 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:47:32.416193 3450 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48364
I20260927 02:47:32.416555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3513
I20260927 02:47:32.420574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3647
I20260927 02:47:32.424247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3781
I20260927 02:47:32.427983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3419
2026-09-27T02:47:32Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1713 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3922](info): set up 2 sockets
Sep 27 02:47:32 dist-test-slave-f4rg krb5kdc[3922](info): commencing operation
krb5kdc: starting...
W20260927 02:47:33.386360 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:33 dist-test-slave-f4rg krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:33.493692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:33.572295 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:33.572587 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:33.572623 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:33.573792 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:33.573832 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:33.573851 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:33.573866 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:33.573889 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:33.575363 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:33.575611 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:33.575824 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:33.580327 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:33.580355 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:33.580399 3935 server_base.cc:1034] running on GCE node
W20260927 02:47:33.580355 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:33.580770 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:33.581025 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:33.582161 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790477253582136 us; error 41 us; skew 500 ppm
Sep 27 02:47:33 dist-test-slave-f4rg krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:33.584631 3935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:33.585047 3935 webserver.cc:460] Webserver started at http://127.0.98.62:39355/ using document root <none> and password file <none>
I20260927 02:47:33.585260 3935 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:33.585304 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:33.585428 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:33.586462 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "05f4c15d166042ab81a07a975ba83772"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "b5b4d5d6a6cc29a17d6e689438615ce9"
server_key_iv: "25f8684fceaba4fe4ef612fecff113b0"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.586774 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "05f4c15d166042ab81a07a975ba83772"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "b5b4d5d6a6cc29a17d6e689438615ce9"
server_key_iv: "25f8684fceaba4fe4ef612fecff113b0"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.588028 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:47:33.588706 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.588864 3935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:33.588939 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "05f4c15d166042ab81a07a975ba83772"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "b5b4d5d6a6cc29a17d6e689438615ce9"
server_key_iv: "25f8684fceaba4fe4ef612fecff113b0"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.589011 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:33.609490 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:33.611306 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:33.611426 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:33.615131 3935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42091
I20260927 02:47:33.615176 4002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42091 every 8 connection(s)
I20260927 02:47:33.615495 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:33.616284 4003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:33.618399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 3935
I20260927 02:47:33.618503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:33.618602 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772: Bootstrap starting.
I20260927 02:47:33.618669 392 external_mini_cluster.cc:1428] Setting key 9f9efffc8ce6038b574442be124b76c3
I20260927 02:47:33.619282 4003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:33.619571 4003 log.cc:824] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:33.620253 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772: No bootstrap required, opened a new log
I20260927 02:47:33.622339 4003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4c15d166042ab81a07a975ba83772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42091 } }
I20260927 02:47:33.622512 4003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:33.622545 4003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4c15d166042ab81a07a975ba83772, State: Initialized, Role: FOLLOWER
I20260927 02:47:33.622725 4003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4c15d166042ab81a07a975ba83772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42091 } }
I20260927 02:47:33.622828 4003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:33.622859 4003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:33.622920 4003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:33.623566 4003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4c15d166042ab81a07a975ba83772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42091 } }
I20260927 02:47:33.623669 4003 leader_election.cc:302] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f4c15d166042ab81a07a975ba83772; no voters:
I20260927 02:47:33.623816 4003 leader_election.cc:288] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:33.623955 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:33.624073 4003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:33.624145 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [term 1 LEADER]: Becoming Leader. State: Replica: 05f4c15d166042ab81a07a975ba83772, State: Running, Role: LEADER
I20260927 02:47:33.624253 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4c15d166042ab81a07a975ba83772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42091 } }
I20260927 02:47:33.624583 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f4c15d166042ab81a07a975ba83772. Latest consensus state: current_term: 1 leader_uuid: "05f4c15d166042ab81a07a975ba83772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4c15d166042ab81a07a975ba83772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42091 } } }
I20260927 02:47:33.624724 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:33.624943 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:33.625403 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:47:33 dist-test-slave-f4rg krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:33.625423 4009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f4c15d166042ab81a07a975ba83772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f4c15d166042ab81a07a975ba83772" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42091 } } }
I20260927 02:47:33.625504 4009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:33.626930 4017 catalog_manager.cc:1347] Generated new cluster ID: 6948694ddc4f4a7ea5b5aa2687954c09
I20260927 02:47:33.626978 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:33.629608 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:33.619165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55456 (local address 127.0.98.62:42091)
0927 02:47:33.619310 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:33.619318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:33.619350 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:47:33.619819 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:33.619828 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:33.619892 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:33.620039 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:33.620762 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.621627 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.622644 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.623282 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.623388 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:33.626171 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:33.626174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:33.626175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:33.626188 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:33.628075 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:33.628429 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:33.628433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:33.628433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:33.628535 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:33.628708 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:33.628709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:33.628709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:33.629015 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:33.629165 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:33.629240 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 02:47:33.629375 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260927 02:47:33.632586 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:33.633260 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:33.651026 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05f4c15d166042ab81a07a975ba83772: Generated new TSK 0
I20260927 02:47:33.651242 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:33.678228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42091
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:33.756896 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:33.757110 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:33.757133 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:33.758258 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:33.758298 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:33.758324 4029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:33.759791 4029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:33.760035 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:33.760250 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:33.764932 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:33.764948 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:33.764948 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:33.764991 4029 server_base.cc:1034] running on GCE node
I20260927 02:47:33.765308 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:33.765561 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:33.766708 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790477253766692 us; error 30 us; skew 500 ppm
Sep 27 02:47:33 dist-test-slave-f4rg krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:33.769172 4029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:33.769599 4029 webserver.cc:460] Webserver started at http://127.0.98.1:35933/ using document root <none> and password file <none>
I20260927 02:47:33.769805 4029 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:33.769852 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:33.769974 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:33.771067 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "f7b9bfad92a740fe8329c2b90aa11580"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "109faebedc3d4459aa87de451b668278"
server_key_iv: "742f5136908ce93b69b5210344ec384b"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.771385 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f7b9bfad92a740fe8329c2b90aa11580"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "109faebedc3d4459aa87de451b668278"
server_key_iv: "742f5136908ce93b69b5210344ec384b"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.772603 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:33.773267 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.773500 4029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:33.773595 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "f7b9bfad92a740fe8329c2b90aa11580"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "109faebedc3d4459aa87de451b668278"
server_key_iv: "742f5136908ce93b69b5210344ec384b"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.773659 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:33.784387 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:33.786165 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:33.786295 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:33.786500 4029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:33.786767 4029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:33.786798 4029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.786850 4029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:33.786873 4029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.792330 4029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42337
I20260927 02:47:33.792375 4157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42337 every 8 connection(s)
I20260927 02:47:33.792698 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:47:33 dist-test-slave-f4rg krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:33.800979 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:33.793356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42327 (local address 127.0.98.62:42091)
0927 02:47:33.793417 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:33.793419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:33.793632 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:47:33.794515 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:33.794516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:33.794526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:33.794582 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:33.795224 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.795748 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.796669 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.796854 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.796900 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:33.798901 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:33.798903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:33.798903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:33.798909 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:33.799963 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:33.800296 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:33.800297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:33.800297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:33.800338 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:33.800521 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:33.800521 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:33.800521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:33.800616 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:33.800660 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:33.800858 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 02:47:33.800916 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:47:33.801151 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:33.793338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42091 (local address 127.0.98.1:42327)
0927 02:47:33.793504 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:33.793545 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:33.794436 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:33.794642 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:33.794645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:33.794960 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:33.795159 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:33.795163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.795815 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.795815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:33.796600 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:33.796601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.796908 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.796908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:33.797000 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:33.797478 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:33.797486 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:33.798782 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:33.800025 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:33.800030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:33.800032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:33.800242 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:33.800379 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:33.800379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:33.800380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:33.800466 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:33.800657 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:33.800659 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:33.800768 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:47:33.800843 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:47:33.800955 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260927 02:47:33.801589 4158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42091
I20260927 02:47:33.801692 4158 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:33.801859 4158 heartbeater.cc:511] Master 127.0.98.62:42091 requested a full tablet report, sending...
I20260927 02:47:33.802304 3967 ts_manager.cc:194] Registered new tserver with Master: f7b9bfad92a740fe8329c2b90aa11580 (127.0.98.1:42337)
I20260927 02:47:33.802928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4029
I20260927 02:47:33.802951 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42327
I20260927 02:47:33.803014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:33.803202 392 external_mini_cluster.cc:1428] Setting key 3ab58494f6176e7380adf46f314ca852
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:33.825037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42091
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:33.903213 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:33.903425 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:33.903448 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:33.904640 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:33.904685 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:33.904711 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:33.906188 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:33.906406 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:33.906621 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:33.911437 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:33.911579 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:33.911542 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:33.911633 4163 server_base.cc:1034] running on GCE node
I20260927 02:47:33.911903 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:33.912133 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:33.913287 4163 hybrid_clock.cc:648] HybridClock initialized: now 1790477253913265 us; error 37 us; skew 500 ppm
Sep 27 02:47:33 dist-test-slave-f4rg krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:33.915809 4163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:33.916256 4163 webserver.cc:460] Webserver started at http://127.0.98.2:39451/ using document root <none> and password file <none>
I20260927 02:47:33.916457 4163 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:33.916505 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:33.916627 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:33.917654 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c4959f6e25b49eaaf127d328a368491"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "814b55a0ea1110de4bbcfa90889e1aec"
server_key_iv: "4c7e01c8b606fd5217f0bb9df2b2bd60"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.917963 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c4959f6e25b49eaaf127d328a368491"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "814b55a0ea1110de4bbcfa90889e1aec"
server_key_iv: "4c7e01c8b606fd5217f0bb9df2b2bd60"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.919210 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:47:33.919906 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.920058 4163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.920154 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "4c4959f6e25b49eaaf127d328a368491"
format_stamp: "Formatted at 2026-09-27 02:47:33 on dist-test-slave-f4rg"
server_key: "814b55a0ea1110de4bbcfa90889e1aec"
server_key_iv: "4c7e01c8b606fd5217f0bb9df2b2bd60"
server_key_version: "encryptionkey@0"
I20260927 02:47:33.920213 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:33.930244 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:33.932862 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:33.933009 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:33.933236 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:33.933568 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:33.933620 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.933665 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:33.933683 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:33.939287 4163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36049
I20260927 02:47:33.939352 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36049 every 8 connection(s)
I20260927 02:47:33.939643 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:47:33 dist-test-slave-f4rg krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:33.948225 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:33.940317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45173 (local address 127.0.98.62:42091)
0927 02:47:33.940374 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:33.940376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:33.940569 (+ 193us) server_negotiation.cc:407] Connection header received
0927 02:47:33.941456 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:33.941457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:33.941467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:33.941526 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:33.942135 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.942639 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.943619 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.943805 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.943858 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:33.945759 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:33.945760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:33.945760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:33.945767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:33.947043 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:33.947458 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:33.947459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:33.947460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:33.947518 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:33.947716 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:33.947716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:33.947716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:33.947816 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:33.947862 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:33.948091 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:47:33.948159 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:47:33.948360 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:33.940295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42091 (local address 127.0.98.2:45173)
0927 02:47:33.940447 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:33.940488 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:33.941368 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:33.941582 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:33.941586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:33.941844 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:33.942032 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:33.942037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.942714 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.942715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:33.943549 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:33.943551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:33.943869 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:33.943869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:33.943971 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:33.944388 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:33.944396 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:33.945679 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:33.947109 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:33.947115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:33.947117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:33.947350 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:33.947575 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:33.947575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:33.947575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:33.947670 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:33.947866 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:33.947870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:33.947987 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 02:47:33.948072 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:47:33.948182 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260927 02:47:33.948822 4292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42091
I20260927 02:47:33.948904 4292 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:33.949044 4292 heartbeater.cc:511] Master 127.0.98.62:42091 requested a full tablet report, sending...
I20260927 02:47:33.949394 3967 ts_manager.cc:194] Registered new tserver with Master: 4c4959f6e25b49eaaf127d328a368491 (127.0.98.2:36049)
I20260927 02:47:33.949394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4163
I20260927 02:47:33.949504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:33.949695 392 external_mini_cluster.cc:1428] Setting key ab617f8ac03b3af46196d0baa2b430c6
I20260927 02:47:33.949851 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45173
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:33.971977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42091
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:34.052546 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:34.052757 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:34.052778 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:34.053905 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:34.053941 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:34.053966 4297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:34.055542 4297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:34.055739 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:34.055956 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:34.060719 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:34.060767 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:34.060756 4297 server_base.cc:1034] running on GCE node
W20260927 02:47:34.060720 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:34.061138 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:34.061383 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:34.062553 4297 hybrid_clock.cc:648] HybridClock initialized: now 1790477254062536 us; error 32 us; skew 500 ppm
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:34.065075 4297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:34.065485 4297 webserver.cc:460] Webserver started at http://127.0.98.3:43395/ using document root <none> and password file <none>
I20260927 02:47:34.065687 4297 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:34.065728 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:34.065846 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:34.067140 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb9315e5e9e341b0a712ab4e37f3468c"
format_stamp: "Formatted at 2026-09-27 02:47:34 on dist-test-slave-f4rg"
server_key: "fbb6e9db94eee49ed3cc53e3562423b5"
server_key_iv: "cc5f59b49977b0d37abf3d514cd018a3"
server_key_version: "encryptionkey@0"
I20260927 02:47:34.067499 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb9315e5e9e341b0a712ab4e37f3468c"
format_stamp: "Formatted at 2026-09-27 02:47:34 on dist-test-slave-f4rg"
server_key: "fbb6e9db94eee49ed3cc53e3562423b5"
server_key_iv: "cc5f59b49977b0d37abf3d514cd018a3"
server_key_version: "encryptionkey@0"
I20260927 02:47:34.068701 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 02:47:34.069396 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:34.069550 4297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:34.069623 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "bb9315e5e9e341b0a712ab4e37f3468c"
format_stamp: "Formatted at 2026-09-27 02:47:34 on dist-test-slave-f4rg"
server_key: "fbb6e9db94eee49ed3cc53e3562423b5"
server_key_iv: "cc5f59b49977b0d37abf3d514cd018a3"
server_key_version: "encryptionkey@0"
I20260927 02:47:34.069689 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:34.099138 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:34.101326 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:34.101475 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:34.101711 4297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:34.102005 4297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:34.102039 4297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:34.102070 4297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:34.102087 4297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:34.107475 4297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41583
I20260927 02:47:34.107542 4425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41583 every 8 connection(s)
I20260927 02:47:34.107839 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:34.116351 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:34.108581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34865 (local address 127.0.98.62:42091)
0927 02:47:34.108660 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:34.108662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:34.108822 (+ 160us) server_negotiation.cc:407] Connection header received
0927 02:47:34.109714 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:34.109715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:34.109726 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:34.109779 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:34.110348 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.110908 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.111837 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.112015 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.112060 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:34.114049 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:34.114050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:34.114050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:34.114057 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:34.115244 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:34.115609 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:34.115609 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:34.115610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:34.115680 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:34.115866 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:34.115866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:34.115866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:34.115967 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:34.116010 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:34.116224 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 02:47:34.116285 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 02:47:34.116562 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:34.108545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42091 (local address 127.0.98.3:34865)
0927 02:47:34.108684 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:34.108721 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:34.109620 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:34.109837 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:34.109840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:34.110092 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:34.110284 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:34.110288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.110969 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.110970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:34.111774 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:34.111776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.112069 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.112070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:34.112161 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:34.112664 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:34.112673 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:34.113965 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:34.115318 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:34.115323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:34.115325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:34.115534 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:34.115729 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:34.115730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:34.115730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:34.115824 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:34.116007 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:34.116010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:34.116131 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 02:47:34.116208 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:47:34.116330 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260927 02:47:34.116811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4297
I20260927 02:47:34.116925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:34.117033 4426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42091
I20260927 02:47:34.117112 392 external_mini_cluster.cc:1428] Setting key d19cc3f1bec4ceb4f9e679c97c0e099f
I20260927 02:47:34.117129 4426 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:34.117321 4426 heartbeater.cc:511] Master 127.0.98.62:42091 requested a full tablet report, sending...
I20260927 02:47:34.117712 3967 ts_manager.cc:194] Registered new tserver with Master: bb9315e5e9e341b0a712ab4e37f3468c (127.0.98.3:41583)
I20260927 02:47:34.118021 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:47:34.118192 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34865
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:34.144232 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:34.132567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.0.98.1:42337)
0927 02:47:34.132714 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:34.132719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:34.132729 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:47:34.132755 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:34.132757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:34.132771 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:34.132835 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:34.133326 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.134020 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.134738 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.135121 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.135176 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:34.137242 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:34.137250 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:34.137250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:34.137257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:34.143130 (+ 5873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:34.143473 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:34.143475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:34.143476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:34.143539 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:34.143736 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:34.143737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:34.143737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:34.143872 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:34.143981 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:34.144084 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:47:34.144152 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:34.151710 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:34.145517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.0.98.62:42091)
0927 02:47:34.145573 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:34.145575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:34.145582 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:47:34.145602 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:34.145602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:34.145611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:34.145652 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:34.146096 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.146589 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.147298 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:34.147482 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:34.147530 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:34.149560 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:34.149561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:34.149561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:34.149569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:34.150645 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:34.150985 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:34.150987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:34.150987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:34.151060 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:34.151238 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:34.151238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:34.151239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:34.151384 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:34.151533 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:34.151574 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 02:47:34.151642 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 02:47:34.151919 3967 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55460
I20260927 02:47:34.152242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4029
I20260927 02:47:34.156446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4163
I20260927 02:47:34.160305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4297
I20260927 02:47:34.164004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 3935
2026-09-27T02:47:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1735 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[4438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[4438](info): set up 2 sockets
Sep 27 02:47:34 dist-test-slave-f4rg krb5kdc[4438](info): commencing operation
krb5kdc: starting...
W20260927 02:47:35.106563 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:35.218334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:35.298333 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:35.298552 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:35.298575 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:35.299717 4451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:35.299754 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:35.299767 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:35.299779 4451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:35.299789 4451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:35.301209 4451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:35.301403 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:35.301591 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:35.306257 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:35.306298 4451 server_base.cc:1034] running on GCE node
W20260927 02:47:35.306231 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:35.306221 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:35.306691 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:35.307000 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:35.308161 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790477255308146 us; error 30 us; skew 500 ppm
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:35.314075 4451 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:35.314409 4451 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:35.314458 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:35.314586 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:35.315737 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "af3613fa719f4e54aca35c612d1130a0"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "7388f46423d4eaba30e244d2ded102f2"
server_key_iv: "420d95565461b169e3c660aa37b215bf"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.316080 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "af3613fa719f4e54aca35c612d1130a0"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "7388f46423d4eaba30e244d2ded102f2"
server_key_iv: "420d95565461b169e3c660aa37b215bf"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.317320 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:47:35.317998 4465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.318161 4451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:35.318248 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "af3613fa719f4e54aca35c612d1130a0"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "7388f46423d4eaba30e244d2ded102f2"
server_key_iv: "420d95565461b169e3c660aa37b215bf"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.318315 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:35.334697 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:35.336616 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:35.336763 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:35.340334 4451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39937
I20260927 02:47:35.340359 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39937 every 8 connection(s)
I20260927 02:47:35.340636 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:35.341522 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:35.343015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4451
I20260927 02:47:35.343115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:35.343291 392 external_mini_cluster.cc:1428] Setting key 59a2de4e09fec0901ac86ef8f4fb28d8
I20260927 02:47:35.344107 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0: Bootstrap starting.
I20260927 02:47:35.344708 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:35.345021 4518 log.cc:824] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:35.345703 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0: No bootstrap required, opened a new log
I20260927 02:47:35.347685 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3613fa719f4e54aca35c612d1130a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260927 02:47:35.347821 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:35.347841 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3613fa719f4e54aca35c612d1130a0, State: Initialized, Role: FOLLOWER
I20260927 02:47:35.347956 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3613fa719f4e54aca35c612d1130a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260927 02:47:35.348038 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:35.348068 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:35.348114 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:35.348759 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3613fa719f4e54aca35c612d1130a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260927 02:47:35.348872 4518 leader_election.cc:302] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af3613fa719f4e54aca35c612d1130a0; no voters:
I20260927 02:47:35.349044 4518 leader_election.cc:288] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:35.349085 4523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:35.349217 4523 raft_consensus.cc:686] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [term 1 LEADER]: Becoming Leader. State: Replica: af3613fa719f4e54aca35c612d1130a0, State: Running, Role: LEADER
I20260927 02:47:35.349288 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:35.349309 4523 consensus_queue.cc:237] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3613fa719f4e54aca35c612d1130a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } }
I20260927 02:47:35.349733 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af3613fa719f4e54aca35c612d1130a0. Latest consensus state: current_term: 1 leader_uuid: "af3613fa719f4e54aca35c612d1130a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3613fa719f4e54aca35c612d1130a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } } }
I20260927 02:47:35.349807 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:35.349972 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af3613fa719f4e54aca35c612d1130a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af3613fa719f4e54aca35c612d1130a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39937 } } }
I20260927 02:47:35.350030 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0 [sys.catalog]: This master's current role is: LEADER
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:35.350131 4530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:35.350795 4530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:35.352471 4530 catalog_manager.cc:1347] Generated new cluster ID: eb3997d08c1a4b40947e9dbb733de263
I20260927 02:47:35.352524 4530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:35.354192 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.343847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.0.98.62:39937)
0927 02:47:35.344006 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.344016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.344040 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:47:35.344443 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.344451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.344501 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.344616 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:35.345374 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.346369 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.347424 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.347824 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.347925 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.350575 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:35.350578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:35.350578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:35.350590 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:35.352749 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.353090 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.353093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.353094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.353194 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.353377 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.353377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.353378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.353614 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:35.353739 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.353851 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.353984 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260927 02:47:35.368839 4530 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:35.369397 4530 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:35.383548 4530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0: Generated new TSK 0
I20260927 02:47:35.383756 4530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:35.419821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:35.507036 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:35.507231 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:35.507253 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:35.508399 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:35.508430 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:35.508455 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:35.509894 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:35.510087 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:35.510258 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:35.514909 4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:35.514909 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:35.514912 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:35.514935 4544 server_base.cc:1034] running on GCE node
I20260927 02:47:35.515333 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:35.515588 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:35.516727 4544 hybrid_clock.cc:648] HybridClock initialized: now 1790477255516713 us; error 32 us; skew 500 ppm
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:35.519147 4544 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:35.519481 4544 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:35.519532 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:35.519649 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:35.520641 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "08d3100bbc8a4aafa89d2e0d65af43a6"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "7e2a905428e8ddb5d64a53b8b09d2572"
server_key_iv: "481c8523cac3a2afa5f5c2e53fbc1f55"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.520956 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08d3100bbc8a4aafa89d2e0d65af43a6"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "7e2a905428e8ddb5d64a53b8b09d2572"
server_key_iv: "481c8523cac3a2afa5f5c2e53fbc1f55"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.522121 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:47:35.522771 4558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.522931 4544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.523010 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "08d3100bbc8a4aafa89d2e0d65af43a6"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "7e2a905428e8ddb5d64a53b8b09d2572"
server_key_iv: "481c8523cac3a2afa5f5c2e53fbc1f55"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.523067 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:35.541967 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:35.544124 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:35.544263 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:35.544474 4544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:35.544767 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:35.544799 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.544827 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:35.544852 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.550336 4544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33551
I20260927 02:47:35.550382 4671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33551 every 8 connection(s)
I20260927 02:47:35.550630 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:35.556419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4544
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:35.556553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:35.556815 392 external_mini_cluster.cc:1428] Setting key 5400ba7e02c2f79ffc6079929ab70f58
I20260927 02:47:35.561170 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.551297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40269 (local address 127.0.98.62:39937)
0927 02:47:35.551372 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.551374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.551589 (+ 215us) server_negotiation.cc:407] Connection header received
0927 02:47:35.552046 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.552047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.552057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.552112 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:35.553002 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.553514 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.554486 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.554682 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.554737 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.557110 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:35.557111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:35.557112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:35.557119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:35.559618 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.560107 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.560108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.560109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.560167 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.560504 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.560505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.560505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.560631 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:35.560685 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.561038 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.561106 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 02:47:35.561429 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.551283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39937 (local address 127.0.98.1:40269)
0927 02:47:35.551439 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.551476 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.551939 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.552176 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.552179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.552427 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:35.552916 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.552921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.553590 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.553591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.554393 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.554395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.554784 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.554786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.554961 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.555462 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:35.555470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:35.557021 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:35.559702 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:35.559710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:35.559712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:35.560017 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:35.560239 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:35.560240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:35.560240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:35.560381 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:35.560702 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:35.560708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:35.560898 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.561017 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.561185 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260927 02:47:35.561981 4672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39937
I20260927 02:47:35.562065 4672 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:35.562206 4672 heartbeater.cc:511] Master 127.0.98.62:39937 requested a full tablet report, sending...
I20260927 02:47:35.562686 4482 ts_manager.cc:194] Registered new tserver with Master: 08d3100bbc8a4aafa89d2e0d65af43a6 (127.0.98.1:33551)
I20260927 02:47:35.563304 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40269
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:35.579627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:35.658640 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:35.658866 4677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:35.658901 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:35.660059 4677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:35.660101 4677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:35.660126 4677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:35.661587 4677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:35.661808 4677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:35.662024 4677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:35.666750 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:35.666765 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:35.666765 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:35.666944 4677 server_base.cc:1034] running on GCE node
I20260927 02:47:35.667179 4677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:35.667418 4677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:35.668570 4677 hybrid_clock.cc:648] HybridClock initialized: now 1790477255668551 us; error 39 us; skew 500 ppm
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:35.671149 4677 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:35.671624 4677 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:35.671694 4677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:35.671804 4677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:35.672834 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "32fc7142354e49319c83f240f4beaf93"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "507b6fa932b68979b8933dacdcca4c3f"
server_key_iv: "bbb0f73f1379b2e81fe3310e0ce45395"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.673128 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32fc7142354e49319c83f240f4beaf93"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "507b6fa932b68979b8933dacdcca4c3f"
server_key_iv: "bbb0f73f1379b2e81fe3310e0ce45395"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.674461 4677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:35.675172 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.675328 4677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.675420 4677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "32fc7142354e49319c83f240f4beaf93"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "507b6fa932b68979b8933dacdcca4c3f"
server_key_iv: "bbb0f73f1379b2e81fe3310e0ce45395"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.675479 4677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:35.693432 4677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:35.695353 4677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:35.695472 4677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:35.695644 4677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:35.695951 4677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:35.695983 4677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.696003 4677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:35.696014 4677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.701265 4677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45691
I20260927 02:47:35.701325 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45691 every 8 connection(s)
I20260927 02:47:35.701570 4677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:35.704008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4677
I20260927 02:47:35.704113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:35.704300 392 external_mini_cluster.cc:1428] Setting key 7a514583189ca35392b91786f6e06615
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:35.711238 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.702223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52525 (local address 127.0.98.62:39937)
0927 02:47:35.702304 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.702306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.702484 (+ 178us) server_negotiation.cc:407] Connection header received
0927 02:47:35.702993 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.702994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.703006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.703077 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:35.703980 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.704730 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.705873 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.706135 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.706213 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.708631 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:35.708633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:35.708634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:35.708640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:35.709944 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.710306 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.710307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.710308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.710367 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.710592 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.710594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.710594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.710702 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:35.710757 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.711070 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.711148 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:47:35.711467 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.702193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39937 (local address 127.0.98.2:52525)
0927 02:47:35.702349 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.702396 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.702898 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.703136 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.703139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.703396 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:35.703857 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.703872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.704842 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.704843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.705770 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.705772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.706233 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.706233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.706380 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.707045 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:35.707055 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:35.708540 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:35.710022 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:35.710028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:35.710030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:35.710237 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:35.710427 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:35.710428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:35.710429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:35.710531 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:35.710760 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:35.710767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:35.710943 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.711035 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.711195 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:47:35.711871 4805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39937
I20260927 02:47:35.711974 4805 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:35.712118 4805 heartbeater.cc:511] Master 127.0.98.62:39937 requested a full tablet report, sending...
I20260927 02:47:35.712543 4482 ts_manager.cc:194] Registered new tserver with Master: 32fc7142354e49319c83f240f4beaf93 (127.0.98.2:45691)
I20260927 02:47:35.712921 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:35.729360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:35.808064 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:35.808276 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:35.808308 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:35.809504 4810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:35.809546 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:35.809581 4810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:35.811120 4810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:35.811357 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:35.811568 4810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:35.816241 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:35.816264 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:35.816267 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:35.816569 4810 server_base.cc:1034] running on GCE node
I20260927 02:47:35.816752 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:35.816974 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:35.818115 4810 hybrid_clock.cc:648] HybridClock initialized: now 1790477255818103 us; error 29 us; skew 500 ppm
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:35.820539 4810 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:35.820859 4810 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:35.820909 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:35.821056 4810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:35.822120 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "91cdb764620d4ddde370f514e17d4d89"
server_key_iv: "2497803937b85a984be8641f75a16a59"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.822441 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "91cdb764620d4ddde370f514e17d4d89"
server_key_iv: "2497803937b85a984be8641f75a16a59"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.823711 4810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:35.824347 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.824506 4810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:35.824585 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee"
format_stamp: "Formatted at 2026-09-27 02:47:35 on dist-test-slave-f4rg"
server_key: "91cdb764620d4ddde370f514e17d4d89"
server_key_iv: "2497803937b85a984be8641f75a16a59"
server_key_version: "encryptionkey@0"
I20260927 02:47:35.824644 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:35.832057 4810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:35.833811 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:35.833940 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:35.834184 4810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:35.834479 4810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:35.834511 4810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.834538 4810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:35.834563 4810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:35.839865 4810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34757
I20260927 02:47:35.839915 4937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34757 every 8 connection(s)
I20260927 02:47:35.840150 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:35.843793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4810
I20260927 02:47:35.843909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:35.844089 392 external_mini_cluster.cc:1428] Setting key bbe79d4e482767f7c95adf3ecb5767a3
Sep 27 02:47:35 dist-test-slave-f4rg krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:35.848734 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.840811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51309 (local address 127.0.98.62:39937)
0927 02:47:35.840890 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.840893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.841038 (+ 145us) server_negotiation.cc:407] Connection header received
0927 02:47:35.841465 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.841466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.841475 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.841524 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:35.842355 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.842877 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.843817 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.844110 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.844173 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.846313 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:35.846314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:35.846314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:35.846320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:35.847616 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.847996 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.847997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.847997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.848041 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.848274 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.848274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.848274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.848363 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:35.848406 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.848619 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.848672 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:47:35.848939 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.840782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39937 (local address 127.0.98.3:51309)
0927 02:47:35.840922 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.840956 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.841394 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.841583 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.841586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.841825 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:35.842289 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.842294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.842952 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.842953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.843740 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.843742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.844209 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.844210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.844345 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.844839 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:35.844849 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:35.846235 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:35.847700 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:35.847706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:35.847708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:35.847934 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:35.848114 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:35.848115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:35.848115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:35.848219 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:35.848411 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:35.848414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:35.848523 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.848600 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.848716 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260927 02:47:35.849359 4938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39937
I20260927 02:47:35.849512 4938 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:35.849644 4938 heartbeater.cc:511] Master 127.0.98.62:39937 requested a full tablet report, sending...
I20260927 02:47:35.849931 4482 ts_manager.cc:194] Registered new tserver with Master: e02e0b0c0bae4ff680a2df79b93fe8ee (127.0.98.3:34757)
I20260927 02:47:35.850247 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51309
I20260927 02:47:35.855660 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:47:35.862558 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.856900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.98.62:39937)
0927 02:47:35.856963 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.856965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.857028 (+ 63us) server_negotiation.cc:407] Connection header received
0927 02:47:35.857099 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.857100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.857110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.857154 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:35.857728 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.858214 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.858954 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.859140 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.859193 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.860284 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:35.860285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:35.860285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:35.860293 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:35.861441 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.861767 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.861768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.861768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.861821 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:35.861994 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:35.861995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:35.861996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:35.862139 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:35.862225 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.862328 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.862459 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:47:35.864053 4482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:35.865579 4482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:35.871557 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.867643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33551 (local address 127.0.0.1:54114)
0927 02:47:35.867764 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.867804 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.867868 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.868260 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.868263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.868265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:35.868333 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.868335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.869305 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.869306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.870330 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.870332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.871247 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.871248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.871336 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.871350 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.871431 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.871481 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260927 02:47:35.871575 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.867958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.0.98.1:33551)
0927 02:47:35.868123 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.868128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.868136 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:35.868166 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.868168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.868179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.868244 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:35.868386 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.869212 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.870421 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.871169 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.871232 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.871292 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.871457 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.871493 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260927 02:47:35.871922 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.867822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45691 (local address 127.0.0.1:34636)
0927 02:47:35.867988 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.867995 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.868027 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.868575 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.868576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.868578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:35.868606 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.868607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.869924 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.869925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.870876 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.870878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.871751 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.871752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.871806 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.871809 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.871839 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.871876 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":50,"thread_start_us":36,"threads_started":1}
I20260927 02:47:35.872131 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.867644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34757 (local address 127.0.0.1:43548)
0927 02:47:35.867888 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.867896 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.867936 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.868471 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.868472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.868474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:35.868529 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.868531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.869931 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.869932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.871086 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.871088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.871940 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.871941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.872001 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.872005 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.872033 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.872068 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":25,"thread_start_us":59,"threads_started":1}
I20260927 02:47:35.872192 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.868133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.0.98.2:45691)
0927 02:47:35.868271 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.868276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.868287 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:47:35.868326 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.868328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.868423 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.868489 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:35.868687 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.869826 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.870952 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.871540 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.871599 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.872051 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.872075 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.872118 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260927 02:47:35.872488 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.867975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.0.98.3:34757)
0927 02:47:35.868304 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.868307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.868316 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:35.868347 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.868349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.868362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.868600 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:35.868670 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.869800 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.871180 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.871844 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.872277 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.872341 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.872365 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.872415 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":76,"threads_started":1}
I20260927 02:47:35.872433 4606 tablet_service.cc:1467] Processing CreateTablet for tablet b3b4a6ded1444809956ffa9b71b0444d (DEFAULT_TABLE table=test-table [id=1ae8853631c64333b14e8beffc121438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:35.872784 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b4a6ded1444809956ffa9b71b0444d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:35.872752 4739 tablet_service.cc:1467] Processing CreateTablet for tablet b3b4a6ded1444809956ffa9b71b0444d (DEFAULT_TABLE table=test-table [id=1ae8853631c64333b14e8beffc121438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:35.873073 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b4a6ded1444809956ffa9b71b0444d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:35.873046 4872 tablet_service.cc:1467] Processing CreateTablet for tablet b3b4a6ded1444809956ffa9b71b0444d (DEFAULT_TABLE table=test-table [id=1ae8853631c64333b14e8beffc121438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:35.873338 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b4a6ded1444809956ffa9b71b0444d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:35.875388 4952 tablet_bootstrap.cc:492] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6: Bootstrap starting.
I20260927 02:47:35.875988 4952 tablet_bootstrap.cc:654] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:35.876168 4953 tablet_bootstrap.cc:492] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Bootstrap starting.
I20260927 02:47:35.876670 4952 log.cc:824] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:35.876762 4953 tablet_bootstrap.cc:654] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:35.877283 4953 log.cc:824] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:35.877445 4952 tablet_bootstrap.cc:492] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6: No bootstrap required, opened a new log
I20260927 02:47:35.877511 4952 ts_tablet_manager.cc:1397] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:35.877628 4954 tablet_bootstrap.cc:492] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Bootstrap starting.
I20260927 02:47:35.877998 4953 tablet_bootstrap.cc:492] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: No bootstrap required, opened a new log
I20260927 02:47:35.878057 4953 ts_tablet_manager.cc:1397] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:35.879314 4954 tablet_bootstrap.cc:654] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:35.879528 4952 raft_consensus.cc:348] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.879714 4952 raft_consensus.cc:374] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:35.879762 4952 raft_consensus.cc:729] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d3100bbc8a4aafa89d2e0d65af43a6, State: Initialized, Role: FOLLOWER
I20260927 02:47:35.879827 4954 log.cc:824] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:35.879895 4952 consensus_queue.cc:260] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.879891 4953 raft_consensus.cc:348] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.880044 4953 raft_consensus.cc:374] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:35.880084 4953 raft_consensus.cc:729] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fc7142354e49319c83f240f4beaf93, State: Initialized, Role: FOLLOWER
I20260927 02:47:35.880187 4952 ts_tablet_manager.cc:1428] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:35.880229 4672 heartbeater.cc:503] Master 127.0.98.62:39937 was elected leader, sending a full tablet report...
I20260927 02:47:35.880441 4953 consensus_queue.cc:260] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.880558 4954 tablet_bootstrap.cc:492] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: No bootstrap required, opened a new log
I20260927 02:47:35.880621 4954 ts_tablet_manager.cc:1397] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:47:35.880716 4953 ts_tablet_manager.cc:1428] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:35.880847 4805 heartbeater.cc:503] Master 127.0.98.62:39937 was elected leader, sending a full tablet report...
I20260927 02:47:35.882258 4954 raft_consensus.cc:348] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.882385 4954 raft_consensus.cc:374] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:35.882405 4954 raft_consensus.cc:729] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02e0b0c0bae4ff680a2df79b93fe8ee, State: Initialized, Role: FOLLOWER
I20260927 02:47:35.882481 4954 consensus_queue.cc:260] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.882692 4954 ts_tablet_manager.cc:1428] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:35.882725 4938 heartbeater.cc:503] Master 127.0.98.62:39937 was elected leader, sending a full tablet report...
I20260927 02:47:35.907821 4960 raft_consensus.cc:482] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:35.907951 4960 raft_consensus.cc:504] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.908264 4960 leader_election.cc:288] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08d3100bbc8a4aafa89d2e0d65af43a6 (127.0.98.1:33551), 32fc7142354e49319c83f240f4beaf93 (127.0.98.2:45691)
I20260927 02:47:35.911046 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.908495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33551 (local address 127.0.98.3:47815)
0927 02:47:35.908577 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.908589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.908632 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.908805 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.908806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.908808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:35.908868 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.908870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.909481 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.909482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.910214 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.910216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.910759 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.910759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.910879 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.910883 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.910933 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.910971 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 02:47:35.911087 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.908650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47815 (local address 127.0.98.1:33551)
0927 02:47:35.908697 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.908699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.908708 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:35.908729 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.908730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.908739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.908783 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:35.908936 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.909404 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.910298 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.910686 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.910751 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.910761 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.910945 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.911009 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:47:35.911441 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.908590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35687 (local address 127.0.98.2:45691)
0927 02:47:35.908633 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.908635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.908807 (+ 172us) server_negotiation.cc:407] Connection header received
0927 02:47:35.908841 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.908842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.908855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.908902 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:35.909043 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.909519 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.910423 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.910931 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.910998 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.911008 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.911340 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.911380 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:47:35.911550 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b4a6ded1444809956ffa9b71b0444d" candidate_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" is_pre_election: true
I20260927 02:47:35.911713 4626 raft_consensus.cc:2454] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e02e0b0c0bae4ff680a2df79b93fe8ee in term 0.
I20260927 02:47:35.911883 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.908509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45691 (local address 127.0.98.3:35687)
0927 02:47:35.908723 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.908733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:35.908793 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:35.908903 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:35.908904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:35.908905 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:35.908972 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.908973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.909592 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.909593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.910334 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:35.910335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.911043 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.911044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:35.911257 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.911262 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:47:35.911703 (+ 441us) client_negotiation.cc:240] Negotiation successful
0927 02:47:35.911745 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":60,"thread_start_us":104,"threads_started":1}
I20260927 02:47:35.911942 4825 leader_election.cc:302] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d3100bbc8a4aafa89d2e0d65af43a6, e02e0b0c0bae4ff680a2df79b93fe8ee; no voters:
I20260927 02:47:35.912375 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b4a6ded1444809956ffa9b71b0444d" candidate_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fc7142354e49319c83f240f4beaf93" is_pre_election: true
I20260927 02:47:35.912529 4759 raft_consensus.cc:2454] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e02e0b0c0bae4ff680a2df79b93fe8ee in term 0.
I20260927 02:47:35.917434 4960 raft_consensus.cc:2781] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:35.917519 4960 raft_consensus.cc:482] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:35.917539 4960 raft_consensus.cc:3037] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:35.918373 4960 raft_consensus.cc:504] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.918520 4960 leader_election.cc:288] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [CANDIDATE]: Term 1 election: Requested vote from peers 08d3100bbc8a4aafa89d2e0d65af43a6 (127.0.98.1:33551), 32fc7142354e49319c83f240f4beaf93 (127.0.98.2:45691)
I20260927 02:47:35.918679 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b4a6ded1444809956ffa9b71b0444d" candidate_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fc7142354e49319c83f240f4beaf93"
I20260927 02:47:35.918733 4626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b4a6ded1444809956ffa9b71b0444d" candidate_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6"
I20260927 02:47:35.918776 4759 raft_consensus.cc:3037] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:35.918833 4626 raft_consensus.cc:3037] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:35.919675 4759 raft_consensus.cc:2454] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e02e0b0c0bae4ff680a2df79b93fe8ee in term 1.
I20260927 02:47:35.919675 4626 raft_consensus.cc:2454] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e02e0b0c0bae4ff680a2df79b93fe8ee in term 1.
I20260927 02:47:35.919863 4825 leader_election.cc:302] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d3100bbc8a4aafa89d2e0d65af43a6, e02e0b0c0bae4ff680a2df79b93fe8ee; no voters:
I20260927 02:47:35.920047 4960 raft_consensus.cc:2781] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:35.920192 4960 raft_consensus.cc:686] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 1 LEADER]: Becoming Leader. State: Replica: e02e0b0c0bae4ff680a2df79b93fe8ee, State: Running, Role: LEADER
I20260927 02:47:35.920303 4960 consensus_queue.cc:237] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } }
I20260927 02:47:35.921115 4481 catalog_manager.cc:5495] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee reported cstate change: term changed from 0 to 1, leader changed from <none> to e02e0b0c0bae4ff680a2df79b93fe8ee (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e02e0b0c0bae4ff680a2df79b93fe8ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:35.933624 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.931531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.0.98.3:34757)
0927 02:47:35.931591 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:35.931593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:35.931600 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:47:35.931628 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:35.931629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:35.931644 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:35.931697 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:35.931785 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.932228 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.932934 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:35.933131 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:35.933194 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:35.933285 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:35.933435 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:35.933494 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:35.933511 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 02:47:35.933528 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:47:35.935457 4759 raft_consensus.cc:1261] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 1 FOLLOWER]: Refusing update from remote peer e02e0b0c0bae4ff680a2df79b93fe8ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:35.935544 4626 raft_consensus.cc:1261] T b3b4a6ded1444809956ffa9b71b0444d P 08d3100bbc8a4aafa89d2e0d65af43a6 [term 1 FOLLOWER]: Refusing update from remote peer e02e0b0c0bae4ff680a2df79b93fe8ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:35.935794 4960 consensus_queue.cc:1035] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "32fc7142354e49319c83f240f4beaf93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:35.935949 4962 consensus_queue.cc:1035] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "08d3100bbc8a4aafa89d2e0d65af43a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:35.938457 4967 mvcc.cc:205] Tried to move back new op lower bound from 7333794840309248000 to 7333794840250298368. Current Snapshot: MvccSnapshot[applied={T|T < 7333794840309248000}]
I20260927 02:47:35.940016 4481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:35.940114 4481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:35.940814 4481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af3613fa719f4e54aca35c612d1130a0: Sending DeleteTablet for 3 replicas of tablet b3b4a6ded1444809956ffa9b71b0444d
I20260927 02:47:35.941118 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b4a6ded1444809956ffa9b71b0444d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54114
I20260927 02:47:35.941120 4739 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b4a6ded1444809956ffa9b71b0444d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34636
I20260927 02:47:35.941309 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b4a6ded1444809956ffa9b71b0444d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43548
I20260927 02:47:35.941501 4976 tablet_replica.cc:330] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: stopping tablet replica
I20260927 02:47:35.941505 4974 tablet_replica.cc:330] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: stopping tablet replica
I20260927 02:47:35.941515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4544
I20260927 02:47:35.941594 4974 raft_consensus.cc:2229] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:35.941677 4976 raft_consensus.cc:2229] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:35.941699 4974 raft_consensus.cc:2258] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:35.941826 4976 raft_consensus.cc:2258] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:35.942363 4974 ts_tablet_manager.cc:1905] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:35.942399 4976 ts_tablet_manager.cc:1905] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:35.944094 4976 ts_tablet_manager.cc:1918] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:35.944105 4974 ts_tablet_manager.cc:1918] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:35.944165 4974 log.cc:1196] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/b3b4a6ded1444809956ffa9b71b0444d
I20260927 02:47:35.944252 4976 log.cc:1196] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/b3b4a6ded1444809956ffa9b71b0444d
I20260927 02:47:35.944423 4974 ts_tablet_manager.cc:1939] T b3b4a6ded1444809956ffa9b71b0444d P 32fc7142354e49319c83f240f4beaf93: Deleting consensus metadata
I20260927 02:47:35.944620 4976 ts_tablet_manager.cc:1939] T b3b4a6ded1444809956ffa9b71b0444d P e02e0b0c0bae4ff680a2df79b93fe8ee: Deleting consensus metadata
I20260927 02:47:35.944820 4468 catalog_manager.cc:4841] TS 32fc7142354e49319c83f240f4beaf93 (127.0.98.2:45691): tablet b3b4a6ded1444809956ffa9b71b0444d (table test-table [id=1ae8853631c64333b14e8beffc121438]) successfully deleted
I20260927 02:47:35.945015 4468 catalog_manager.cc:4841] TS e02e0b0c0bae4ff680a2df79b93fe8ee (127.0.98.3:34757): tablet b3b4a6ded1444809956ffa9b71b0444d (table test-table [id=1ae8853631c64333b14e8beffc121438]) successfully deleted
W20260927 02:47:35.947919 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33551 (error 108)
I20260927 02:47:35.948097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4677
I20260927 02:47:35.948333 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:35.948231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33551 (local address 127.0.0.1:54130)
0927 02:47:35.948275 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:35.948301 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:47:35.948571 4466 catalog_manager.cc:4584] TS 08d3100bbc8a4aafa89d2e0d65af43a6 (127.0.98.1:33551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3b4a6ded1444809956ffa9b71b0444d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33551: connect: Connection refused (error 111)
I20260927 02:47:35.953017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4810
I20260927 02:47:35.957165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4451
2026-09-27T02:47:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1793 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T02:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:35Z Disabled control of system clock
I20260927 02:47:35.981992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38823
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:47:36.059450 4981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:36.059655 4981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:36.059674 4981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:36.060806 4981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:36.060842 4981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:36.060853 4981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:36.060864 4981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:36.060874 4981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:36.062170 4981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:36.062358 4981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:36.062589 4981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:47:36.067273 4981 server_base.cc:1034] running on GCE node
W20260927 02:47:36.067245 4987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.067252 4989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.067232 4986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:36.067714 4981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:36.067994 4981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:36.069139 4981 hybrid_clock.cc:648] HybridClock initialized: now 1790477256069121 us; error 31 us; skew 500 ppm
I20260927 02:47:36.070531 4981 webserver.cc:460] Webserver started at http://127.0.98.62:42765/ using document root <none> and password file <none>
I20260927 02:47:36.070741 4981 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:36.070788 4981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:36.070940 4981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:36.071973 4981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "7667fbccf7c04a84a89c682e97a74ad4"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "81cc4c7994e1f2464c4b6cd66a517d57"
server_key_iv: "1ae3f55e53f4548ab3fc0fdb1f301968"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.072316 4981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "7667fbccf7c04a84a89c682e97a74ad4"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "81cc4c7994e1f2464c4b6cd66a517d57"
server_key_iv: "1ae3f55e53f4548ab3fc0fdb1f301968"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.073613 4981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:36.074240 4995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.074415 4981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:36.074497 4981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "7667fbccf7c04a84a89c682e97a74ad4"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "81cc4c7994e1f2464c4b6cd66a517d57"
server_key_iv: "1ae3f55e53f4548ab3fc0fdb1f301968"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.074592 4981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:36.077766 4981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:36.078006 4981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:36.078120 4981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:36.081624 4981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38823
I20260927 02:47:36.081655 5047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38823 every 8 connection(s)
I20260927 02:47:36.081944 4981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:36.082541 5048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:36.085187 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4: Bootstrap starting.
I20260927 02:47:36.085763 5048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:36.085745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 4981
I20260927 02:47:36.085839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:36.086035 392 external_mini_cluster.cc:1428] Setting key abe66653becbd86c666146fc407b577d
I20260927 02:47:36.086066 5048 log.cc:824] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:36.086735 5048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4: No bootstrap required, opened a new log
I20260927 02:47:36.088480 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.086534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.0.98.62:38823)
0927 02:47:36.086731 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.086740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.086771 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:47:36.087265 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.087274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.087322 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.087493 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.088055 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.088059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.088059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.088063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.088109 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.088124 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.088201 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.088231 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.088284 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260927 02:47:36.088753 5048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7667fbccf7c04a84a89c682e97a74ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38823 } }
I20260927 02:47:36.088912 5048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:36.088970 5048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7667fbccf7c04a84a89c682e97a74ad4, State: Initialized, Role: FOLLOWER
I20260927 02:47:36.089080 5048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7667fbccf7c04a84a89c682e97a74ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38823 } }
I20260927 02:47:36.089164 5048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:36.089192 5048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:36.089237 5048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:36.090029 5048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7667fbccf7c04a84a89c682e97a74ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38823 } }
I20260927 02:47:36.090119 5048 leader_election.cc:302] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7667fbccf7c04a84a89c682e97a74ad4; no voters:
I20260927 02:47:36.090261 5048 leader_election.cc:288] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:36.090289 5053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:36.090422 5053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [term 1 LEADER]: Becoming Leader. State: Replica: 7667fbccf7c04a84a89c682e97a74ad4, State: Running, Role: LEADER
I20260927 02:47:36.090472 5048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:36.090533 5053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7667fbccf7c04a84a89c682e97a74ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38823 } }
I20260927 02:47:36.090866 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7667fbccf7c04a84a89c682e97a74ad4. Latest consensus state: current_term: 1 leader_uuid: "7667fbccf7c04a84a89c682e97a74ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7667fbccf7c04a84a89c682e97a74ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38823 } } }
I20260927 02:47:36.090947 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:36.090878 5055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7667fbccf7c04a84a89c682e97a74ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7667fbccf7c04a84a89c682e97a74ad4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38823 } } }
I20260927 02:47:36.091001 5055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:36.091136 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:36.091636 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:36.093050 5060 catalog_manager.cc:1347] Generated new cluster ID: da5f0746fdc24911afc834860c125950
I20260927 02:47:36.093103 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:36.105543 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:36.105723 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:36.108908 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4: Generated new TSK 0
I20260927 02:47:36.109095 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:47:36.113541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38823
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 02:47:36.192468 5072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:36.192659 5072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:36.192688 5072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:36.193813 5072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:36.193849 5072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:36.193882 5072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:36.195225 5072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:36.195462 5072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:36.195727 5072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:36.200354 5077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.200408 5078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.200502 5080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:36.200670 5072 server_base.cc:1034] running on GCE node
I20260927 02:47:36.200852 5072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:36.201067 5072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:36.202221 5072 hybrid_clock.cc:648] HybridClock initialized: now 1790477256202202 us; error 32 us; skew 500 ppm
I20260927 02:47:36.203662 5072 webserver.cc:460] Webserver started at http://127.0.98.1:39305/ using document root <none> and password file <none>
I20260927 02:47:36.203912 5072 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:36.203955 5072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:36.204077 5072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:36.205101 5072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "8069d209d6fc4111803e1018b3d67e78"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "dc960d961c56685b93be3da2d5ad210c"
server_key_iv: "bc879002b3153a7b2717e2893f644e40"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.205420 5072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8069d209d6fc4111803e1018b3d67e78"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "dc960d961c56685b93be3da2d5ad210c"
server_key_iv: "bc879002b3153a7b2717e2893f644e40"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.206641 5072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:36.207377 5086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.207523 5072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.207600 5072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "8069d209d6fc4111803e1018b3d67e78"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "dc960d961c56685b93be3da2d5ad210c"
server_key_iv: "bc879002b3153a7b2717e2893f644e40"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.207669 5072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:36.210711 5072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:36.210992 5072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:36.211115 5072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:36.211305 5072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:36.211570 5072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:36.211601 5072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.211627 5072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:36.211645 5072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.216603 5072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42907
I20260927 02:47:36.216674 5199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42907 every 8 connection(s)
I20260927 02:47:36.216952 5072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:36.218088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5072
I20260927 02:47:36.218166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:36.218319 392 external_mini_cluster.cc:1428] Setting key f6bc27bc367c4271b9941788ff870b26
I20260927 02:47:36.219113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38823
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 02:47:36.220018 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.217685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55741 (local address 127.0.98.62:38823)
0927 02:47:36.217749 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.217752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.218011 (+ 259us) server_negotiation.cc:407] Connection header received
0927 02:47:36.218548 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.218549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.218556 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.218607 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.219780 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.219781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.219781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.219785 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.219800 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.219804 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.219851 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.219938 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.219952 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:47:36.220197 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.217641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38823 (local address 127.0.98.1:55741)
0927 02:47:36.217850 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.217892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.218448 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.218676 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.218680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.218689 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.219631 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.219631 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.219660 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.219688 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.219859 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.219864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.219868 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.219938 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.219998 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260927 02:47:36.220623 5200 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38823
I20260927 02:47:36.220716 5200 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:36.220857 5200 heartbeater.cc:511] Master 127.0.98.62:38823 requested a full tablet report, sending...
I20260927 02:47:36.221355 5012 ts_manager.cc:194] Registered new tserver with Master: 8069d209d6fc4111803e1018b3d67e78 (127.0.98.1:42907)
W20260927 02:47:36.296839 5203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:36.297035 5203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:36.297055 5203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:36.298149 5203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:36.298182 5203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:36.298204 5203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:36.299520 5203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:36.299710 5203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:36.299942 5203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:36.304663 5209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.304679 5208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:36.304813 5203 server_base.cc:1034] running on GCE node
W20260927 02:47:36.304703 5211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:36.305087 5203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:36.305323 5203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:36.306497 5203 hybrid_clock.cc:648] HybridClock initialized: now 1790477256306483 us; error 28 us; skew 500 ppm
I20260927 02:47:36.307731 5203 webserver.cc:460] Webserver started at http://127.0.98.2:44643/ using document root <none> and password file <none>
I20260927 02:47:36.307945 5203 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:36.307993 5203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:36.308116 5203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:36.309160 5203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ce824d9e7894b73ae73a49c7e987a56"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "7b33412c8d6ec8852633cac2c7d4b09b"
server_key_iv: "b18f1d34dc1fe1ace818b0e5ede48530"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.309499 5203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ce824d9e7894b73ae73a49c7e987a56"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "7b33412c8d6ec8852633cac2c7d4b09b"
server_key_iv: "b18f1d34dc1fe1ace818b0e5ede48530"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.310721 5203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:36.311432 5217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.311604 5203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:36.311683 5203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "0ce824d9e7894b73ae73a49c7e987a56"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "7b33412c8d6ec8852633cac2c7d4b09b"
server_key_iv: "b18f1d34dc1fe1ace818b0e5ede48530"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.311748 5203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:36.314895 5203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:36.315129 5203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:36.315245 5203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:36.315426 5203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:36.315691 5203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:36.315726 5203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.315752 5203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:36.315778 5203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.320960 5203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37309
I20260927 02:47:36.321002 5330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37309 every 8 connection(s)
I20260927 02:47:36.321287 5203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:36.323570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5203
I20260927 02:47:36.323657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:36.323807 392 external_mini_cluster.cc:1428] Setting key 51196b06a744e2af0c19e0e8edfe9ab1
I20260927 02:47:36.324138 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.321987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56691 (local address 127.0.98.62:38823)
0927 02:47:36.322072 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.322075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.322263 (+ 188us) server_negotiation.cc:407] Connection header received
0927 02:47:36.322745 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.322746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.322752 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.322822 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.323901 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.323902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.323903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.323907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.323923 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.323927 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.323978 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.324055 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.324069 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:47:36.324312 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.321945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38823 (local address 127.0.98.2:56691)
0927 02:47:36.322147 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.322180 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.322657 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.322899 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.322902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.322911 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.323761 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.323762 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.323788 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.323815 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.323983 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.323988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.323991 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.324057 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.324116 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260927 02:47:36.324779 5331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38823
I20260927 02:47:36.324800 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38823
--builtin_ntp_servers=127.0.98.20:42641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 02:47:36.324852 5331 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:36.324981 5331 heartbeater.cc:511] Master 127.0.98.62:38823 requested a full tablet report, sending...
I20260927 02:47:36.325316 5012 ts_manager.cc:194] Registered new tserver with Master: 0ce824d9e7894b73ae73a49c7e987a56 (127.0.98.2:37309)
W20260927 02:47:36.404532 5334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:36.404734 5334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:36.404762 5334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:36.405884 5334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:36.405927 5334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:36.405961 5334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:36.407348 5334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:36.407572 5334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:36.407837 5334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:36.412612 5342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.412613 5340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.412696 5339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:36.412762 5334 server_base.cc:1034] running on GCE node
I20260927 02:47:36.413014 5334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:36.413230 5334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:36.414392 5334 hybrid_clock.cc:648] HybridClock initialized: now 1790477256414373 us; error 29 us; skew 500 ppm
I20260927 02:47:36.415725 5334 webserver.cc:460] Webserver started at http://127.0.98.3:36725/ using document root <none> and password file <none>
I20260927 02:47:36.415951 5334 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:36.416002 5334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:36.416127 5334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:36.417186 5334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "62fcb1d4413a49f180c60da861a8b348"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "a6d5f1412311275aa26c2f39a2a487bd"
server_key_iv: "73e0a0a69137380c74e3500419346e0f"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.417548 5334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62fcb1d4413a49f180c60da861a8b348"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "a6d5f1412311275aa26c2f39a2a487bd"
server_key_iv: "73e0a0a69137380c74e3500419346e0f"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.418776 5334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:36.419467 5348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.419625 5334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:36.419703 5334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "62fcb1d4413a49f180c60da861a8b348"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "a6d5f1412311275aa26c2f39a2a487bd"
server_key_iv: "73e0a0a69137380c74e3500419346e0f"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.419768 5334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:36.423193 5334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:36.423442 5334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:36.423561 5334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:36.423736 5334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:36.424010 5334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:36.424042 5334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.424068 5334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:36.424093 5334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.429275 5334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38881
I20260927 02:47:36.429309 5461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38881 every 8 connection(s)
I20260927 02:47:36.429615 5334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:36.432348 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.430239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33939 (local address 127.0.98.62:38823)
0927 02:47:36.430301 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.430303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.430517 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:47:36.431033 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.431034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.431040 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.431090 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.432114 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.432116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.432116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.432120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.432135 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.432139 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.432190 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.432266 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.432280 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:47:36.432523 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.430223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38823 (local address 127.0.98.3:33939)
0927 02:47:36.430394 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.430424 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.430926 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.431156 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.431161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.431170 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.431982 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.431983 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.432008 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.432033 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.432198 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.432202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.432205 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.432274 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.432332 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260927 02:47:36.432964 5462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38823
I20260927 02:47:36.433035 5462 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:36.433141 5462 heartbeater.cc:511] Master 127.0.98.62:38823 requested a full tablet report, sending...
I20260927 02:47:36.433427 5012 ts_manager.cc:194] Registered new tserver with Master: 62fcb1d4413a49f180c60da861a8b348 (127.0.98.3:38881)
I20260927 02:47:36.439179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5334
I20260927 02:47:36.439266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:36.439435 392 external_mini_cluster.cc:1428] Setting key 8cffdb6b093b0d7088460513888ead97
I20260927 02:47:36.440318 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:47:36.442513 5052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.441564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.98.62:38823)
0927 02:47:36.441624 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.441625 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.441722 (+ 97us) server_negotiation.cc:407] Connection header received
0927 02:47:36.441762 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.441763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.441768 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.441813 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.442323 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.442324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.442325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.442328 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.442340 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.442344 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.442385 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.442432 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.442443 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:47:36.443944 5012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:36.445436 5012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:36.448930 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.447686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.98.2:37309)
0927 02:47:36.447959 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.447964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.447973 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:36.447998 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.448000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.448009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.448231 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.448619 (+ 388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.448625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.448625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.448628 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.448660 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.448679 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.448759 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.448824 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.448838 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":69,"threads_started":1}
I20260927 02:47:36.448920 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.447425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42907 (local address 127.0.0.1:34004)
0927 02:47:36.447653 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.447692 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.447749 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.447972 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.447973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.447976 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.448402 (+ 426us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.448403 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.448420 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.448457 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.448774 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.448775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.448776 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.448803 (+ 27us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.448817 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":65,"threads_started":1}
I20260927 02:47:36.448920 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.447465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37309 (local address 127.0.0.1:36512)
0927 02:47:36.447822 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.447830 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.447857 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.448105 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.448106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.448108 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.448532 (+ 424us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.448533 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.448540 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.448551 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.448753 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.448759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.448762 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.448819 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.448831 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":95,"threads_started":1}
I20260927 02:47:36.449069 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.447495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.0.98.3:38881)
0927 02:47:36.447641 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.447646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.447805 (+ 159us) server_negotiation.cc:407] Connection header received
0927 02:47:36.447832 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.447834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.447846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.447919 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.448763 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.448766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.448766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.448769 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.448794 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.448808 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.448858 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.448974 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.448988 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260927 02:47:36.449167 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.447425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38881 (local address 127.0.0.1:41376)
0927 02:47:36.447697 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.447707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.447757 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.447935 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.447941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.447947 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.448672 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.448673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.448677 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.448686 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.448881 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.448882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.448882 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.448919 (+ 37us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.448941 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260927 02:47:36.449642 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.447658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.0.98.1:42907)
0927 02:47:36.447834 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.447838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.447847 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:36.447870 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.447872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.447880 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.448167 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.448645 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.448649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.448649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.448653 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.448683 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.448699 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.449547 (+ 848us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.449568 (+ 21us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.449580 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260927 02:47:36.449604 5265 tablet_service.cc:1467] Processing CreateTablet for tablet db7efbd1d693442393fc3dd4e4c3603f (DEFAULT_TABLE table=test-table [id=8bc64ac89a204267a479486577e2fe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:36.449937 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7efbd1d693442393fc3dd4e4c3603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:36.450227 5134 tablet_service.cc:1467] Processing CreateTablet for tablet db7efbd1d693442393fc3dd4e4c3603f (DEFAULT_TABLE table=test-table [id=8bc64ac89a204267a479486577e2fe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:36.450531 5134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7efbd1d693442393fc3dd4e4c3603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:36.452561 5476 tablet_bootstrap.cc:492] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Bootstrap starting.
I20260927 02:47:36.452907 5477 tablet_bootstrap.cc:492] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78: Bootstrap starting.
I20260927 02:47:36.453203 5476 tablet_bootstrap.cc:654] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:36.453212 5396 tablet_service.cc:1467] Processing CreateTablet for tablet db7efbd1d693442393fc3dd4e4c3603f (DEFAULT_TABLE table=test-table [id=8bc64ac89a204267a479486577e2fe62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:36.453506 5476 log.cc:824] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:36.453536 5396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7efbd1d693442393fc3dd4e4c3603f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:36.453586 5477 tablet_bootstrap.cc:654] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:36.453871 5477 log.cc:824] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:36.454192 5476 tablet_bootstrap.cc:492] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: No bootstrap required, opened a new log
I20260927 02:47:36.454265 5476 ts_tablet_manager.cc:1397] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:36.454566 5477 tablet_bootstrap.cc:492] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78: No bootstrap required, opened a new log
I20260927 02:47:36.454632 5477 ts_tablet_manager.cc:1397] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:36.456171 5480 tablet_bootstrap.cc:492] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Bootstrap starting.
I20260927 02:47:36.456714 5480 tablet_bootstrap.cc:654] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:36.456772 5476 raft_consensus.cc:348] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.456955 5476 raft_consensus.cc:374] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:36.456977 5480 log.cc:824] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:36.457002 5476 raft_consensus.cc:729] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce824d9e7894b73ae73a49c7e987a56, State: Initialized, Role: FOLLOWER
I20260927 02:47:36.456993 5477 raft_consensus.cc:348] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.457131 5477 raft_consensus.cc:374] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:36.457134 5476 consensus_queue.cc:260] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.457171 5477 raft_consensus.cc:729] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8069d209d6fc4111803e1018b3d67e78, State: Initialized, Role: FOLLOWER
I20260927 02:47:36.457297 5477 consensus_queue.cc:260] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.457389 5476 ts_tablet_manager.cc:1428] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:36.457499 5331 heartbeater.cc:503] Master 127.0.98.62:38823 was elected leader, sending a full tablet report...
I20260927 02:47:36.457542 5477 ts_tablet_manager.cc:1428] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:36.457635 5480 tablet_bootstrap.cc:492] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: No bootstrap required, opened a new log
I20260927 02:47:36.457683 5480 ts_tablet_manager.cc:1397] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:36.457751 5200 heartbeater.cc:503] Master 127.0.98.62:38823 was elected leader, sending a full tablet report...
I20260927 02:47:36.459741 5480 raft_consensus.cc:348] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.459861 5480 raft_consensus.cc:374] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:36.459889 5480 raft_consensus.cc:729] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fcb1d4413a49f180c60da861a8b348, State: Initialized, Role: FOLLOWER
I20260927 02:47:36.460005 5480 consensus_queue.cc:260] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.460224 5480 ts_tablet_manager.cc:1428] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:36.460270 5462 heartbeater.cc:503] Master 127.0.98.62:38823 was elected leader, sending a full tablet report...
W20260927 02:47:36.467518 5201 tablet.cc:2369] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:47:36.504595 5483 raft_consensus.cc:482] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:36.504760 5483 raft_consensus.cc:504] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.505127 5483 leader_election.cc:288] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62fcb1d4413a49f180c60da861a8b348 (127.0.98.3:38881), 0ce824d9e7894b73ae73a49c7e987a56 (127.0.98.2:37309)
I20260927 02:47:36.506389 5202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.505295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38881 (local address 127.0.98.1:56963)
0927 02:47:36.505338 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.505349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.505390 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.505528 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.505529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.505531 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.505999 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.505999 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.506009 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.506022 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.506253 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.506254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.506255 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.506306 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.506320 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 02:47:36.506398 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.505387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56963 (local address 127.0.98.3:38881)
0927 02:47:36.505428 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.505430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.505437 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:47:36.505462 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.505463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.505468 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.505520 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.506139 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.506141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.506141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.506146 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.506164 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.506168 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.506232 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.506316 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.506328 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:47:36.506488 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.505432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55729 (local address 127.0.98.2:37309)
0927 02:47:36.505475 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.505477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.505600 (+ 123us) server_negotiation.cc:407] Connection header received
0927 02:47:36.505626 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.505627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.505632 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.505681 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.506270 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.506271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.506271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.506276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.506292 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.506297 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.506346 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.506407 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.506427 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:47:36.506559 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.505302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37309 (local address 127.0.98.1:55729)
0927 02:47:36.505509 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.505518 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.505561 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.505771 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.505772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.505774 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:36.506154 (+ 380us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:47:36.506155 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:36.506161 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:47:36.506172 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:36.506356 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:36.506357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:36.506357 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 02:47:36.506400 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:47:36.506412 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":20,"thread_start_us":103,"threads_started":1}
I20260927 02:47:36.506769 5416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7efbd1d693442393fc3dd4e4c3603f" candidate_uuid: "8069d209d6fc4111803e1018b3d67e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fcb1d4413a49f180c60da861a8b348" is_pre_election: true
I20260927 02:47:36.506861 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7efbd1d693442393fc3dd4e4c3603f" candidate_uuid: "8069d209d6fc4111803e1018b3d67e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce824d9e7894b73ae73a49c7e987a56" is_pre_election: true
I20260927 02:47:36.506959 5416 raft_consensus.cc:2454] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8069d209d6fc4111803e1018b3d67e78 in term 0.
I20260927 02:47:36.507033 5285 raft_consensus.cc:2454] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8069d209d6fc4111803e1018b3d67e78 in term 0.
I20260927 02:47:36.507213 5088 leader_election.cc:302] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fcb1d4413a49f180c60da861a8b348, 8069d209d6fc4111803e1018b3d67e78; no voters:
I20260927 02:47:36.507355 5483 raft_consensus.cc:2781] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:36.507409 5483 raft_consensus.cc:482] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:36.507431 5483 raft_consensus.cc:3037] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:36.508139 5483 raft_consensus.cc:504] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.508257 5483 leader_election.cc:288] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [CANDIDATE]: Term 1 election: Requested vote from peers 62fcb1d4413a49f180c60da861a8b348 (127.0.98.3:38881), 0ce824d9e7894b73ae73a49c7e987a56 (127.0.98.2:37309)
I20260927 02:47:36.508389 5416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7efbd1d693442393fc3dd4e4c3603f" candidate_uuid: "8069d209d6fc4111803e1018b3d67e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fcb1d4413a49f180c60da861a8b348"
I20260927 02:47:36.508459 5416 raft_consensus.cc:3037] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:36.508482 5285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7efbd1d693442393fc3dd4e4c3603f" candidate_uuid: "8069d209d6fc4111803e1018b3d67e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ce824d9e7894b73ae73a49c7e987a56"
I20260927 02:47:36.508565 5285 raft_consensus.cc:3037] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:36.509130 5416 raft_consensus.cc:2454] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8069d209d6fc4111803e1018b3d67e78 in term 1.
I20260927 02:47:36.509254 5285 raft_consensus.cc:2454] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8069d209d6fc4111803e1018b3d67e78 in term 1.
I20260927 02:47:36.509328 5088 leader_election.cc:302] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fcb1d4413a49f180c60da861a8b348, 8069d209d6fc4111803e1018b3d67e78; no voters:
I20260927 02:47:36.509404 5483 raft_consensus.cc:2781] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:36.509541 5483 raft_consensus.cc:686] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [term 1 LEADER]: Becoming Leader. State: Replica: 8069d209d6fc4111803e1018b3d67e78, State: Running, Role: LEADER
I20260927 02:47:36.509626 5483 consensus_queue.cc:237] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } }
I20260927 02:47:36.510445 5011 catalog_manager.cc:5495] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8069d209d6fc4111803e1018b3d67e78 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8069d209d6fc4111803e1018b3d67e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8069d209d6fc4111803e1018b3d67e78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42907 } health_report { overall_health: HEALTHY } } }
I20260927 02:47:36.522957 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.521796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.0.98.1:42907)
0927 02:47:36.521855 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.521857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.521865 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:36.521950 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.521951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.521964 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.522010 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.522695 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.522695 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.522696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.522701 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.522717 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.522721 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.522769 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.522869 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.522882 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:47:36.524745 5285 raft_consensus.cc:1261] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 1 FOLLOWER]: Refusing update from remote peer 8069d209d6fc4111803e1018b3d67e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:36.524745 5416 raft_consensus.cc:1261] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 1 FOLLOWER]: Refusing update from remote peer 8069d209d6fc4111803e1018b3d67e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:36.525117 5486 consensus_queue.cc:1035] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ce824d9e7894b73ae73a49c7e987a56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:36.525262 5483 consensus_queue.cc:1035] T db7efbd1d693442393fc3dd4e4c3603f P 8069d209d6fc4111803e1018b3d67e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fcb1d4413a49f180c60da861a8b348" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:36.529549 5012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:36.529632 5012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:36.530288 5012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7667fbccf7c04a84a89c682e97a74ad4: Sending DeleteTablet for 3 replicas of tablet db7efbd1d693442393fc3dd4e4c3603f
I20260927 02:47:36.530568 5265 tablet_service.cc:1514] Processing DeleteTablet for tablet db7efbd1d693442393fc3dd4e4c3603f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:36 UTC) from {username='slave'} at 127.0.0.1:36512
I20260927 02:47:36.530614 5134 tablet_service.cc:1514] Processing DeleteTablet for tablet db7efbd1d693442393fc3dd4e4c3603f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:36 UTC) from {username='slave'} at 127.0.0.1:34004
I20260927 02:47:36.530709 5396 tablet_service.cc:1514] Processing DeleteTablet for tablet db7efbd1d693442393fc3dd4e4c3603f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:36 UTC) from {username='slave'} at 127.0.0.1:41376
I20260927 02:47:36.530929 5501 tablet_replica.cc:330] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: stopping tablet replica
I20260927 02:47:36.530946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5072
I20260927 02:47:36.531029 5501 raft_consensus.cc:2229] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:36.531121 5501 raft_consensus.cc:2258] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:36.531245 5499 tablet_replica.cc:330] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: stopping tablet replica
I20260927 02:47:36.531322 5499 raft_consensus.cc:2229] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:36.531690 5499 raft_consensus.cc:2258] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:36.532105 5501 ts_tablet_manager.cc:1905] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:36.532454 5499 ts_tablet_manager.cc:1905] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:36.533890 5501 ts_tablet_manager.cc:1918] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:36.533901 5499 ts_tablet_manager.cc:1918] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:36.533962 5501 log.cc:1196] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/db7efbd1d693442393fc3dd4e4c3603f
I20260927 02:47:36.533977 5499 log.cc:1196] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/db7efbd1d693442393fc3dd4e4c3603f
I20260927 02:47:36.534199 5501 ts_tablet_manager.cc:1939] T db7efbd1d693442393fc3dd4e4c3603f P 62fcb1d4413a49f180c60da861a8b348: Deleting consensus metadata
I20260927 02:47:36.534214 5499 ts_tablet_manager.cc:1939] T db7efbd1d693442393fc3dd4e4c3603f P 0ce824d9e7894b73ae73a49c7e987a56: Deleting consensus metadata
I20260927 02:47:36.534578 4998 catalog_manager.cc:4841] TS 0ce824d9e7894b73ae73a49c7e987a56 (127.0.98.2:37309): tablet db7efbd1d693442393fc3dd4e4c3603f (table test-table [id=8bc64ac89a204267a479486577e2fe62]) successfully deleted
I20260927 02:47:36.534596 4997 catalog_manager.cc:4841] TS 62fcb1d4413a49f180c60da861a8b348 (127.0.98.3:38881): tablet db7efbd1d693442393fc3dd4e4c3603f (table test-table [id=8bc64ac89a204267a479486577e2fe62]) successfully deleted
W20260927 02:47:36.536691 4999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42907 (error 108)
I20260927 02:47:36.537177 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.537057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42907 (local address 127.0.0.1:34012)
0927 02:47:36.537107 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.537146 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:47:36.537213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5203
W20260927 02:47:36.537313 4999 catalog_manager.cc:4584] TS 8069d209d6fc4111803e1018b3d67e78 (127.0.98.1:42907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db7efbd1d693442393fc3dd4e4c3603f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42907: connect: Connection refused (error 111)
I20260927 02:47:36.541180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5334
I20260927 02:47:36.544836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 4981
2026-09-27T02:47:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (587 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 02:47:36.562832 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:47:36.563990 392 webserver.cc:460] Webserver started at https://127.0.0.1:35869/ using document root <none> and password file <none>
I20260927 02:47:36.564095 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:47:36.564234 392 webserver.cc:460] Webserver started at http://127.0.0.1:33055/ using document root <none> and password file <none>
2026-09-27T02:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:36Z Disabled control of system clock
I20260927 02:47:36.571391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33025
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33025
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35869/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/testchainca.cert with env {}
W20260927 02:47:36.648792 5508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:36.649013 5508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:36.649034 5508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:36.650173 5508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:36.650208 5508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:36.650220 5508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:36.650231 5508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:36.650241 5508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:47:36.650252 5508 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:47:36.650262 5508 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35869/jwks.json
I20260927 02:47:36.651743 5508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35869/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:36.651950 5508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:36.652120 5508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:36.656873 5513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.656909 5516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.657117 5514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:36.657156 5508 server_base.cc:1034] running on GCE node
I20260927 02:47:36.657380 5508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:36.657660 5508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:36.658838 5508 hybrid_clock.cc:648] HybridClock initialized: now 1790477256658804 us; error 42 us; skew 500 ppm
I20260927 02:47:36.660097 5508 webserver.cc:460] Webserver started at http://127.0.98.62:37051/ using document root <none> and password file <none>
I20260927 02:47:36.660312 5508 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:36.660362 5508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:36.660483 5508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:36.661523 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "55c5d40c3edf4e60bdaf4944a2ef546e"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "7698a2d643a629dffcebeebf0387884b"
server_key_iv: "c136f7664339933809d70113f4a58061"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.661854 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "55c5d40c3edf4e60bdaf4944a2ef546e"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "7698a2d643a629dffcebeebf0387884b"
server_key_iv: "c136f7664339933809d70113f4a58061"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.663137 5508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:36.663812 5522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.663968 5508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.664048 5508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "55c5d40c3edf4e60bdaf4944a2ef546e"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "7698a2d643a629dffcebeebf0387884b"
server_key_iv: "c136f7664339933809d70113f4a58061"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.664110 5508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:36.698722 5508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:36.699098 5508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:36.699235 5508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:36.702775 5508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33025
I20260927 02:47:36.702847 5574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33025 every 8 connection(s)
I20260927 02:47:36.703171 5508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:36.704107 5575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:36.705447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5508
I20260927 02:47:36.705534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:36.705685 392 external_mini_cluster.cc:1428] Setting key 5cb288fc698c03f5d6c1c49529ada261
I20260927 02:47:36.706615 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e: Bootstrap starting.
I20260927 02:47:36.707315 5575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:36.707638 5575 log.cc:824] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:36.708318 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e: No bootstrap required, opened a new log
I20260927 02:47:36.710356 5575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33025 } }
I20260927 02:47:36.710534 5575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:36.710572 5575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c5d40c3edf4e60bdaf4944a2ef546e, State: Initialized, Role: FOLLOWER
I20260927 02:47:36.710686 5575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33025 } }
I20260927 02:47:36.710743 5575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:36.710763 5575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:36.710829 5575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:36.711350 5575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33025 } }
I20260927 02:47:36.711476 5575 leader_election.cc:302] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55c5d40c3edf4e60bdaf4944a2ef546e; no voters:
I20260927 02:47:36.711504 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.706184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.0.98.62:33025)
0927 02:47:36.706387 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.706396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.706427 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:47:36.706980 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.706990 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.707075 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.707219 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.707384 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.708610 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.709369 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.710002 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.710134 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:36.710871 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.710876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.710876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.711047 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.711081 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.711091 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.711145 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.711218 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.711293 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260927 02:47:36.711623 5575 leader_election.cc:288] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:36.711658 5580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:36.711782 5580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [term 1 LEADER]: Becoming Leader. State: Replica: 55c5d40c3edf4e60bdaf4944a2ef546e, State: Running, Role: LEADER
I20260927 02:47:36.711856 5575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:36.711865 5580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33025 } }
I20260927 02:47:36.712204 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33025 } } }
I20260927 02:47:36.712267 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:36.712379 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c5d40c3edf4e60bdaf4944a2ef546e. Latest consensus state: current_term: 1 leader_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55c5d40c3edf4e60bdaf4944a2ef546e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33025 } } }
I20260927 02:47:36.712383 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:47:36.712433 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:36.712572 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:36.713009 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:36.714556 5586 catalog_manager.cc:1347] Generated new cluster ID: ea25568d3ba0449eb3356d220b64d523
I20260927 02:47:36.714608 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:36.733590 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:36.734294 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:36.737790 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55c5d40c3edf4e60bdaf4944a2ef546e: Generated new TSK 0
I20260927 02:47:36.737977 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:47:36.769675 5605 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 02:47:36.769781 5605 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:47:36.770092 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.747117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.98.62:33025)
0927 02:47:36.747166 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.747168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.747286 (+ 118us) server_negotiation.cc:407] Connection header received
0927 02:47:36.747386 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.747387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.747396 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.747479 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:47:36.747633 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.748116 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.749212 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.749448 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.749503 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:36.749718 (+ 215us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:47:36.769672 (+ 19954us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:47:36.770016 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60878: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 02:47:36.770176 5578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60878: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 02:47:36.770282 5604 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:36.747039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33025 (local address 127.0.0.1:60878)
0927 02:47:36.747207 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.747219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.747294 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.747504 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.747505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.747507 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:47:36.747573 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:36.747575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.748186 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.748187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:36.749118 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:36.749120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.749520 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.749523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:36.749592 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:36.749613 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:47:36.769818 (+ 20205us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 02:47:36.769946 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33025: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35869/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 02:47:36.770694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5508
2026-09-27T02:47:36Z chronyd exiting
I20260927 02:47:36.805954 392 test_util.cc:175] -----------------------------------------------
I20260927 02:47:36.806039 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790477232139228-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 02:47:36.806771 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:47:36.807456 392 webserver.cc:460] Webserver started at https://127.0.0.1:39709/ using document root <none> and password file <none>
I20260927 02:47:36.807502 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:47:36.807632 392 webserver.cc:460] Webserver started at http://127.0.0.1:38627/ using document root <none> and password file <none>
2026-09-27T02:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:36Z Disabled control of system clock
I20260927 02:47:36.814599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41357
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41357
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39709/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/testchainca.cert with env {}
W20260927 02:47:36.892168 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:36.892377 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:36.892405 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:36.893512 5611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:36.893544 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:36.893555 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:36.893566 5611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:36.893584 5611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:47:36.893599 5611 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:47:36.893615 5611 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39709/jwks.json
I20260927 02:47:36.895119 5611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39709/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:36.895332 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:36.895500 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:47:36.900127 5611 server_base.cc:1034] running on GCE node
W20260927 02:47:36.900082 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.900059 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:36.900063 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:36.900527 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:36.900803 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:36.901942 5611 hybrid_clock.cc:648] HybridClock initialized: now 1790477256901920 us; error 37 us; skew 500 ppm
I20260927 02:47:36.903062 5611 webserver.cc:460] Webserver started at http://127.0.98.62:34533/ using document root <none> and password file <none>
I20260927 02:47:36.903272 5611 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:36.903319 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:36.903435 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:36.904445 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "152dea8ede5644bfab6f4eed54e0392a"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "69f9446589278a33d6ee0aeabb36d30f"
server_key_iv: "d4d7f9c229cc65d2f8e4cda790d60632"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.904773 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "152dea8ede5644bfab6f4eed54e0392a"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "69f9446589278a33d6ee0aeabb36d30f"
server_key_iv: "d4d7f9c229cc65d2f8e4cda790d60632"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.905930 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:36.906527 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:36.906679 5611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:36.906756 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "152dea8ede5644bfab6f4eed54e0392a"
format_stamp: "Formatted at 2026-09-27 02:47:36 on dist-test-slave-f4rg"
server_key: "69f9446589278a33d6ee0aeabb36d30f"
server_key_iv: "d4d7f9c229cc65d2f8e4cda790d60632"
server_key_version: "encryptionkey@0"
I20260927 02:47:36.906847 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:36.916276 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:36.916554 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:36.916675 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:36.920277 5611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41357
I20260927 02:47:36.920305 5677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41357 every 8 connection(s)
I20260927 02:47:36.920605 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:36.921446 5678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:36.923705 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a: Bootstrap starting.
I20260927 02:47:36.924256 5678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:36.924502 5678 log.cc:824] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:36.925118 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a: No bootstrap required, opened a new log
I20260927 02:47:36.926618 5678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152dea8ede5644bfab6f4eed54e0392a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41357 } }
I20260927 02:47:36.926750 5678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:36.926771 5678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152dea8ede5644bfab6f4eed54e0392a, State: Initialized, Role: FOLLOWER
I20260927 02:47:36.926924 5678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152dea8ede5644bfab6f4eed54e0392a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41357 } }
I20260927 02:47:36.926986 5678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:36.927007 5678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:36.927052 5678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:36.927532 5678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152dea8ede5644bfab6f4eed54e0392a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41357 } }
I20260927 02:47:36.927618 5678 leader_election.cc:302] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 152dea8ede5644bfab6f4eed54e0392a; no voters:
I20260927 02:47:36.927760 5678 leader_election.cc:288] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:36.927796 5681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:36.927922 5681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [term 1 LEADER]: Becoming Leader. State: Replica: 152dea8ede5644bfab6f4eed54e0392a, State: Running, Role: LEADER
I20260927 02:47:36.928005 5678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:36.928015 5681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152dea8ede5644bfab6f4eed54e0392a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41357 } }
I20260927 02:47:36.928490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5611
I20260927 02:47:36.928596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:36.928768 392 external_mini_cluster.cc:1428] Setting key 43d36e4fa30da019fcc420c0911cf925
I20260927 02:47:36.929497 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "152dea8ede5644bfab6f4eed54e0392a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152dea8ede5644bfab6f4eed54e0392a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41357 } } }
I20260927 02:47:36.929584 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:36.929809 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152dea8ede5644bfab6f4eed54e0392a. Latest consensus state: current_term: 1 leader_uuid: "152dea8ede5644bfab6f4eed54e0392a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152dea8ede5644bfab6f4eed54e0392a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41357 } } }
I20260927 02:47:36.929867 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a [sys.catalog]: This master's current role is: LEADER
W20260927 02:47:36.930154 5698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:47:36.930209 5698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:47:36.930930 5700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:36.931111 5700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:36.933506 5700 catalog_manager.cc:1347] Generated new cluster ID: 6dd89329ae994207a1cba165368fd061
I20260927 02:47:36.933555 5700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:36.934012 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.929197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.0.98.62:41357)
0927 02:47:36.929508 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.929517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.929549 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:47:36.929668 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.929676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.929744 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.929882 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:36.930076 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.931055 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.932045 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.932503 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.932609 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:36.933277 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:36.933280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:36.933281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:36.933463 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:36.933505 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:36.933520 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:36.933574 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:36.933763 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 02:47:36.933843 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":75,"threads_started":1}
I20260927 02:47:36.934795 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:47:36.939306 5700 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:36.939780 5700 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:36.945513 5700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 152dea8ede5644bfab6f4eed54e0392a: Generated new TSK 0
I20260927 02:47:36.945690 5700 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 02:47:36.967626 5708 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 02:47:36.967713 5708 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 02:47:36.968046 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:36.954259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.0.98.62:41357)
0927 02:47:36.954301 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:36.954302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:36.954415 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:47:36.954490 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:36.954491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:36.954502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:36.954561 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:47:36.954701 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.955272 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.956017 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.956232 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.956285 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:36.956449 (+ 164us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:47:36.967814 (+ 11365us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:47:36.967978 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:47:36.968103 5695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 02:47:36.968107 5707 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:36.954155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41357 (local address 127.0.0.1:53086)
0927 02:47:36.954338 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:36.954349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:36.954419 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:36.954567 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:36.954568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:36.954570 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:47:36.954641 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:36.954643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.955345 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.955346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:36.955939 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:36.955942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:36.956293 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:36.956297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:36.956357 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:36.956362 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:47:36.967925 (+ 11563us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 02:47:36.968024 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41357: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39709/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260927 02:47:36.968415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5611
2026-09-27T02:47:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (198 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 02:47:37.004769 392 mini_oidc.cc:152] Starting JWKS server
I20260927 02:47:37.005419 392 webserver.cc:460] Webserver started at https://127.0.0.1:39529/ using document root <none> and password file <none>
I20260927 02:47:37.005458 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 02:47:37.005564 392 webserver.cc:460] Webserver started at http://127.0.0.1:45157/ using document root <none> and password file <none>
2026-09-27T02:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:37Z Disabled control of system clock
I20260927 02:47:37.012274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40397
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39529/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:47:37.090608 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:37.090867 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:37.090902 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:37.092052 5714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:37.092089 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:37.092105 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:37.092123 5714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:37.092137 5714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:47:37.092154 5714 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:47:37.092173 5714 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39529/jwks.json
I20260927 02:47:37.093528 5714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39529/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:37.093765 5714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:37.094044 5714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:37.098755 5720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:37.098914 5714 server_base.cc:1034] running on GCE node
W20260927 02:47:37.098762 5719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:37.098762 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:37.099275 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:37.099548 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:37.100705 5714 hybrid_clock.cc:648] HybridClock initialized: now 1790477257100694 us; error 27 us; skew 500 ppm
I20260927 02:47:37.101851 5714 webserver.cc:460] Webserver started at http://127.0.98.62:35145/ using document root <none> and password file <none>
I20260927 02:47:37.102067 5714 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:37.102113 5714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:37.102214 5714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:37.103390 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "7826c9cde0634a46a4f9b1cc231f90f0"
format_stamp: "Formatted at 2026-09-27 02:47:37 on dist-test-slave-f4rg"
server_key: "11a25158e7fb7df462740838cb1e31e1"
server_key_iv: "c54a8f9a542edd8e8eafb351abf03824"
server_key_version: "encryptionkey@0"
I20260927 02:47:37.103721 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "7826c9cde0634a46a4f9b1cc231f90f0"
format_stamp: "Formatted at 2026-09-27 02:47:37 on dist-test-slave-f4rg"
server_key: "11a25158e7fb7df462740838cb1e31e1"
server_key_iv: "c54a8f9a542edd8e8eafb351abf03824"
server_key_version: "encryptionkey@0"
I20260927 02:47:37.104964 5714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:37.105685 5728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:37.105896 5714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:37.105993 5714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "7826c9cde0634a46a4f9b1cc231f90f0"
format_stamp: "Formatted at 2026-09-27 02:47:37 on dist-test-slave-f4rg"
server_key: "11a25158e7fb7df462740838cb1e31e1"
server_key_iv: "c54a8f9a542edd8e8eafb351abf03824"
server_key_version: "encryptionkey@0"
I20260927 02:47:37.106057 5714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:37.117250 5714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:37.117539 5714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:37.117664 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:37.121382 5714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40397
I20260927 02:47:37.121415 5780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40397 every 8 connection(s)
I20260927 02:47:37.121719 5714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:37.122318 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:37.124881 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0: Bootstrap starting.
I20260927 02:47:37.125432 5781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:37.125685 5781 log.cc:824] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:37.126106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5714
I20260927 02:47:37.126199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:37.126338 392 external_mini_cluster.cc:1428] Setting key 3b887b72cdd157de485e2212e1341bcb
I20260927 02:47:37.126369 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0: No bootstrap required, opened a new log
I20260927 02:47:37.128321 5781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:47:37.128499 5781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:37.128547 5781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7826c9cde0634a46a4f9b1cc231f90f0, State: Initialized, Role: FOLLOWER
I20260927 02:47:37.128666 5781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:47:37.128737 5781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:37.128767 5781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:37.128813 5781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:37.129338 5781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:47:37.129427 5781 leader_election.cc:302] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7826c9cde0634a46a4f9b1cc231f90f0; no voters:
I20260927 02:47:37.129586 5781 leader_election.cc:288] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:37.129627 5786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:37.129756 5786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [term 1 LEADER]: Becoming Leader. State: Replica: 7826c9cde0634a46a4f9b1cc231f90f0, State: Running, Role: LEADER
I20260927 02:47:37.129822 5781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:37.129840 5786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260927 02:47:37.130143 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7826c9cde0634a46a4f9b1cc231f90f0. Latest consensus state: current_term: 1 leader_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260927 02:47:37.130225 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:37.130153 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7826c9cde0634a46a4f9b1cc231f90f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260927 02:47:37.130366 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:37.130482 5792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:37.130951 5792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:37.132520 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:37.126822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.0.98.62:40397)
0927 02:47:37.127017 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:37.127025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:37.127049 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:47:37.127574 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:37.127581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:37.127638 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:37.127747 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:37.127977 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.129342 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.130457 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.131072 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.131193 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:37.131824 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:37.131829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:37.131829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:37.132018 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:37.132061 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:37.132075 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:37.132126 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:37.132234 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 02:47:37.132311 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260927 02:47:37.133137 5792 catalog_manager.cc:1347] Generated new cluster ID: 1a709d488371456f80f3060d57e086f1
I20260927 02:47:37.133184 5792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:37.133467 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:47:37.141992 5792 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:37.142477 5792 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:37.145684 5792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7826c9cde0634a46a4f9b1cc231f90f0: Generated new TSK 0
I20260927 02:47:37.145865 5792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:47:37.154070 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:37.150802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.0.98.62:40397)
0927 02:47:37.150881 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:37.150883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:37.150922 (+ 39us) server_negotiation.cc:407] Connection header received
0927 02:47:37.150996 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:37.150997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:37.151009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:37.151075 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:47:37.151191 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.151830 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.152615 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.152821 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.152874 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:37.153038 (+ 164us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:47:37.153865 (+ 827us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:47:37.153999 (+ 134us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 02:47:37.154125 5785 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 02:47:37.154187 5810 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:37.150725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40397 (local address 127.0.0.1:39610)
0927 02:47:37.150863 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:37.150873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:37.150940 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:37.151075 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:37.151075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:37.151077 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:47:37.151132 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:37.151134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.151933 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.151934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:37.152537 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:37.152539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.152880 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.152888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:37.152952 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:37.152957 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:47:37.153975 (+ 1018us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 02:47:37.154084 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40397: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39529/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":95,"threads_started":1}
I20260927 02:47:37.154744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5714
E20260927 02:47:37.165026 5811 webserver.cc:533] Webserver: sslize failed:
E20260927 02:47:37.165104 5811 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T02:47:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (186 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T02:47:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:37Z Disabled control of system clock
I20260927 02:47:37.197932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36427
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36427
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/testchainca.cert with env {}
W20260927 02:47:37.276290 5815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:37.276507 5815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:37.276530 5815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:37.277665 5815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:37.277700 5815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:37.277712 5815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:37.277724 5815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:37.277733 5815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 02:47:37.277745 5815 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 02:47:37.277757 5815 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 02:47:37.279229 5815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:37.279430 5815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:37.279600 5815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:37.284205 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:37.284245 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:37.284354 5815 server_base.cc:1034] running on GCE node
W20260927 02:47:37.284438 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:37.284699 5815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:37.284986 5815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:37.286134 5815 hybrid_clock.cc:648] HybridClock initialized: now 1790477257286115 us; error 28 us; skew 500 ppm
I20260927 02:47:37.287390 5815 webserver.cc:460] Webserver started at http://127.0.98.62:44567/ using document root <none> and password file <none>
I20260927 02:47:37.287638 5815 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:37.287689 5815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:37.287815 5815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:37.288857 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "940d959724ce49d382ba13d66928fa0a"
format_stamp: "Formatted at 2026-09-27 02:47:37 on dist-test-slave-f4rg"
server_key: "1bc8ce9e11eeda42839c05113a7e1176"
server_key_iv: "ed64ddf4ed6fc7eeefc6eaa7e5fa7459"
server_key_version: "encryptionkey@0"
I20260927 02:47:37.289182 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "940d959724ce49d382ba13d66928fa0a"
format_stamp: "Formatted at 2026-09-27 02:47:37 on dist-test-slave-f4rg"
server_key: "1bc8ce9e11eeda42839c05113a7e1176"
server_key_iv: "ed64ddf4ed6fc7eeefc6eaa7e5fa7459"
server_key_version: "encryptionkey@0"
I20260927 02:47:37.290408 5815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:47:37.291121 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:37.291332 5815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:37.291410 5815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "940d959724ce49d382ba13d66928fa0a"
format_stamp: "Formatted at 2026-09-27 02:47:37 on dist-test-slave-f4rg"
server_key: "1bc8ce9e11eeda42839c05113a7e1176"
server_key_iv: "ed64ddf4ed6fc7eeefc6eaa7e5fa7459"
server_key_version: "encryptionkey@0"
I20260927 02:47:37.291487 5815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:37.300189 5815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:37.300434 5815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:37.300537 5815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:37.304267 5815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36427
I20260927 02:47:37.304294 5881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36427 every 8 connection(s)
I20260927 02:47:37.304612 5815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:37.305466 5882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:37.307720 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a: Bootstrap starting.
I20260927 02:47:37.308274 5882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:37.308527 5882 log.cc:824] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:37.309159 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a: No bootstrap required, opened a new log
I20260927 02:47:37.310716 5882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940d959724ce49d382ba13d66928fa0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36427 } }
I20260927 02:47:37.310901 5882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:37.310952 5882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940d959724ce49d382ba13d66928fa0a, State: Initialized, Role: FOLLOWER
I20260927 02:47:37.311094 5882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940d959724ce49d382ba13d66928fa0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36427 } }
I20260927 02:47:37.311157 5882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:37.311187 5882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:37.311231 5882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:37.311818 5882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940d959724ce49d382ba13d66928fa0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36427 } }
I20260927 02:47:37.311925 5882 leader_election.cc:302] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 940d959724ce49d382ba13d66928fa0a; no voters:
I20260927 02:47:37.311929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5815
I20260927 02:47:37.312045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:37.312094 5882 leader_election.cc:288] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:37.312180 5885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:37.312230 392 external_mini_cluster.cc:1428] Setting key 31e2e4b43bc4f068a9b62f3b10543b5c
I20260927 02:47:37.312352 5882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:37.312359 5885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [term 1 LEADER]: Becoming Leader. State: Replica: 940d959724ce49d382ba13d66928fa0a, State: Running, Role: LEADER
I20260927 02:47:37.312942 5885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940d959724ce49d382ba13d66928fa0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36427 } }
I20260927 02:47:37.313974 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "940d959724ce49d382ba13d66928fa0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940d959724ce49d382ba13d66928fa0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36427 } } }
I20260927 02:47:37.314098 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:37.314117 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 940d959724ce49d382ba13d66928fa0a. Latest consensus state: current_term: 1 leader_uuid: "940d959724ce49d382ba13d66928fa0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940d959724ce49d382ba13d66928fa0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36427 } } }
I20260927 02:47:37.314196 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:37.314309 5902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:37.314786 5902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:37.316428 5902 catalog_manager.cc:1347] Generated new cluster ID: a8404dd801514b3d8e775f6c1bd2c3d4
I20260927 02:47:37.316484 5902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:37.317891 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:37.312726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.0.98.62:36427)
0927 02:47:37.312941 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:37.312952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:37.312984 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:47:37.313110 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:37.313117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:37.313176 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:37.313303 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:37.313544 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.314913 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.315846 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.316346 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.316467 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:37.317244 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:37.317248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:37.317248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:47:37.317426 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:37.317468 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:47:37.317479 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:37.317531 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:37.317613 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:47:37.317690 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260927 02:47:37.318712 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:47:37.330689 5902 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:37.331434 5902 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:37.345780 5902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 940d959724ce49d382ba13d66928fa0a: Generated new TSK 0
I20260927 02:47:37.346020 5902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:47:37.374209 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:37.353264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.98.62:36427)
0927 02:47:37.353308 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:37.353310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:37.353359 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:47:37.353444 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:37.353445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:37.353458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:37.353513 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0927 02:47:37.353640 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.354149 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.354966 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.355172 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.355226 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:37.355404 (+ 178us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 02:47:37.373993 (+ 18589us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:47:37.374150 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43730: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 02:47:37.374269 5890 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43730: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 02:47:37.374398 5911 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:37.353158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36427 (local address 127.0.0.1:43730)
0927 02:47:37.353303 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:37.353314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:37.353376 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:37.353520 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:37.353521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:37.353523 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 02:47:37.353577 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:37.353579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.354225 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.354226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:37.354884 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:37.354887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:37.355240 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:37.355246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:37.355305 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:37.355311 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 02:47:37.374145 (+ 18834us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 02:47:37.374290 (+ 145us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36427: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260927 02:47:37.374787 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5815
2026-09-27T02:47:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (199 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:37 dist-test-slave-f4rg krb5kdc[5923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:37 dist-test-slave-f4rg krb5kdc[5923](info): set up 2 sockets
Sep 27 02:47:37 dist-test-slave-f4rg krb5kdc[5923](info): commencing operation
krb5kdc: starting...
W20260927 02:47:38.486776 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:38 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:47:38.599257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:38.677356 5936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:38.677562 5936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:38.677582 5936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:38.678719 5936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:38.678759 5936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:38.678772 5936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:38.678783 5936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:38.678799 5936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:38.680335 5936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:38.680536 5936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:38.680706 5936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:38.685238 5941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:38.685302 5944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:38.685228 5942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:38.685263 5936 server_base.cc:1034] running on GCE node
I20260927 02:47:38.685631 5936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:38.685900 5936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:38.687036 5936 hybrid_clock.cc:648] HybridClock initialized: now 1790477258687016 us; error 33 us; skew 500 ppm
Sep 27 02:47:38 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:38.689288 5936 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:47:38.689643 5936 webserver.cc:460] Webserver started at http://127.0.98.62:42025/ using document root <none> and password file <none>
I20260927 02:47:38.689821 5936 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:38.689867 5936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:38.689955 5936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:38.690964 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "b9bcadcc285a43d3b1e50502c673aaed"
format_stamp: "Formatted at 2026-09-27 02:47:38 on dist-test-slave-f4rg"
server_key: "775abf953f77f3c9e9d290514f58b4ce"
server_key_iv: "5b02c391c9437899e492107e3f8db00e"
server_key_version: "encryptionkey@0"
I20260927 02:47:38.691247 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9bcadcc285a43d3b1e50502c673aaed"
format_stamp: "Formatted at 2026-09-27 02:47:38 on dist-test-slave-f4rg"
server_key: "775abf953f77f3c9e9d290514f58b4ce"
server_key_iv: "5b02c391c9437899e492107e3f8db00e"
server_key_version: "encryptionkey@0"
I20260927 02:47:38.692356 5936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:47:38.692974 5951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:38.693140 5936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:38.693213 5936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "b9bcadcc285a43d3b1e50502c673aaed"
format_stamp: "Formatted at 2026-09-27 02:47:38 on dist-test-slave-f4rg"
server_key: "775abf953f77f3c9e9d290514f58b4ce"
server_key_iv: "5b02c391c9437899e492107e3f8db00e"
server_key_version: "encryptionkey@0"
I20260927 02:47:38.693285 5936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:38.704280 5936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:38.706220 5936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:38.706326 5936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:38.709782 5936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34541
I20260927 02:47:38.709813 6003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34541 every 8 connection(s)
I20260927 02:47:38.710263 5936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:38.710903 6004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:38.713029 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed: Bootstrap starting.
I20260927 02:47:38.713565 6004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:38.713609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 5936
I20260927 02:47:38.713696 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:38.713873 392 external_mini_cluster.cc:1428] Setting key 5d7095bf155dd9e3c3f8ba7b65729ee4
I20260927 02:47:38.713860 6004 log.cc:824] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:38.714609 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed: No bootstrap required, opened a new log
I20260927 02:47:38.717056 6004 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bcadcc285a43d3b1e50502c673aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34541 } }
I20260927 02:47:38.717221 6004 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:38.717257 6004 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bcadcc285a43d3b1e50502c673aaed, State: Initialized, Role: FOLLOWER
I20260927 02:47:38.717392 6004 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bcadcc285a43d3b1e50502c673aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34541 } }
I20260927 02:47:38.717468 6004 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:38.717499 6004 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:38.717543 6004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:38.718101 6004 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bcadcc285a43d3b1e50502c673aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34541 } }
I20260927 02:47:38.718202 6004 leader_election.cc:302] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9bcadcc285a43d3b1e50502c673aaed; no voters:
I20260927 02:47:38.718354 6004 leader_election.cc:288] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:38.718400 6009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:38.718550 6009 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [term 1 LEADER]: Becoming Leader. State: Replica: b9bcadcc285a43d3b1e50502c673aaed, State: Running, Role: LEADER
I20260927 02:47:38.718691 6009 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bcadcc285a43d3b1e50502c673aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34541 } }
I20260927 02:47:38.718570 6004 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:38.719090 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9bcadcc285a43d3b1e50502c673aaed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bcadcc285a43d3b1e50502c673aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34541 } } }
I20260927 02:47:38.719302 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:38.719244 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bcadcc285a43d3b1e50502c673aaed. Latest consensus state: current_term: 1 leader_uuid: "b9bcadcc285a43d3b1e50502c673aaed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bcadcc285a43d3b1e50502c673aaed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34541 } } }
I20260927 02:47:38.719542 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:38.719662 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:38.720064 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 02:47:38 dist-test-slave-f4rg krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:38.722271 6013 catalog_manager.cc:1347] Generated new cluster ID: 6523a60f5992431f8106f3014dbf92dd
I20260927 02:47:38.722324 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:47:38.724618 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:38.714301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.0.98.62:34541)
0927 02:47:38.714483 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:38.714493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:38.714523 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:47:38.715045 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:38.715055 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:38.715117 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:38.715262 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:38.715817 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:38.717157 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:38.717968 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:38.718550 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:38.718689 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:38.721174 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:38.721178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:38.721179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:38.721190 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:38.723098 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:38.723503 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:38.723505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:38.723506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:38.723596 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:38.723791 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:38.723791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:38.723792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:38.724055 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:38.724192 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:38.724300 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 02:47:38.724421 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260927 02:47:38.737427 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:38.737922 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:38.740780 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9bcadcc285a43d3b1e50502c673aaed: Generated new TSK 0
I20260927 02:47:38.740944 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:47:38.770525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34541
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:38.849076 6030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:38.849308 6030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:38.849331 6030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:38.850482 6030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:38.850523 6030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:38.850548 6030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:38.852010 6030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:38.852228 6030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:38.852443 6030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:38.857147 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:38.857154 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:38.857182 6030 server_base.cc:1034] running on GCE node
W20260927 02:47:38.857160 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:38.857708 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:38.857944 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:38.859117 6030 hybrid_clock.cc:648] HybridClock initialized: now 1790477258859094 us; error 40 us; skew 500 ppm
Sep 27 02:47:38 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:38.861642 6030 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:47:38.862087 6030 webserver.cc:460] Webserver started at http://127.0.98.1:33263/ using document root <none> and password file <none>
I20260927 02:47:38.862298 6030 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:38.862346 6030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:38.862470 6030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:38.863518 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "30827ed63eb4426dba557c2e1a78e5cf"
format_stamp: "Formatted at 2026-09-27 02:47:38 on dist-test-slave-f4rg"
server_key: "537a8436132d815a688586088830c880"
server_key_iv: "26ac84378bbc1abe7af658502aed67fc"
server_key_version: "encryptionkey@0"
I20260927 02:47:38.863837 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30827ed63eb4426dba557c2e1a78e5cf"
format_stamp: "Formatted at 2026-09-27 02:47:38 on dist-test-slave-f4rg"
server_key: "537a8436132d815a688586088830c880"
server_key_iv: "26ac84378bbc1abe7af658502aed67fc"
server_key_version: "encryptionkey@0"
I20260927 02:47:38.865033 6030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:38.865733 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:38.865919 6030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:38.866007 6030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "30827ed63eb4426dba557c2e1a78e5cf"
format_stamp: "Formatted at 2026-09-27 02:47:38 on dist-test-slave-f4rg"
server_key: "537a8436132d815a688586088830c880"
server_key_iv: "26ac84378bbc1abe7af658502aed67fc"
server_key_version: "encryptionkey@0"
I20260927 02:47:38.866070 6030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:38.878257 6030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:38.880401 6030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:38.880539 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:38.880751 6030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:38.881049 6030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:38.881083 6030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:38.881111 6030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:38.881135 6030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:38.886724 6030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37165
I20260927 02:47:38.886844 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37165 every 8 connection(s)
I20260927 02:47:38.887113 6030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:47:38 dist-test-slave-f4rg krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:38.895076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6030
I20260927 02:47:38.895172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:38.895334 392 external_mini_cluster.cc:1428] Setting key 7950ae1c3907ab7042afac22a21ae2aa
I20260927 02:47:38.896082 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:38.887814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43157 (local address 127.0.98.62:34541)
0927 02:47:38.887907 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:38.887909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:38.888072 (+ 163us) server_negotiation.cc:407] Connection header received
0927 02:47:38.889030 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:38.889031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:38.889042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:38.889089 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:38.889733 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:38.890234 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:38.891244 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:38.891448 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:38.891502 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:38.893586 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:38.893588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:38.893589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:38.893596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:38.894720 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:38.895136 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:38.895137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:38.895138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:38.895199 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:38.895420 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:38.895421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:38.895421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:38.895545 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:38.895605 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:38.895898 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 02:47:38.895987 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 02:47:38.896281 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:38.887744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34541 (local address 127.0.98.1:43157)
0927 02:47:38.887935 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:38.887980 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:38.888918 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:38.889173 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:38.889177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:38.889448 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:38.889638 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:38.889643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:38.890318 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:38.890319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:38.891157 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:38.891159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:38.891520 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:38.891521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:38.891641 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:38.892150 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:38.892160 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:38.893488 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:38.894827 (+ 1339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:38.894834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:38.894836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:38.895068 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:38.895255 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:38.895255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:38.895256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:38.895361 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:38.895615 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:38.895621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:38.895772 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 02:47:38.895867 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:47:38.896052 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260927 02:47:38.896709 6159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34541
I20260927 02:47:38.896806 6159 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:38.896982 6159 heartbeater.cc:511] Master 127.0.98.62:34541 requested a full tablet report, sending...
I20260927 02:47:38.897490 5968 ts_manager.cc:194] Registered new tserver with Master: 30827ed63eb4426dba557c2e1a78e5cf (127.0.98.1:37165)
I20260927 02:47:38.898329 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43157
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:47:38.917761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34541
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:38.996632 6164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:38.996855 6164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:38.996888 6164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:38.998034 6164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:38.998073 6164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:38.998107 6164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:38.999593 6164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:38.999825 6164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:39.000029 6164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:39.004791 6170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:39.004801 6164 server_base.cc:1034] running on GCE node
W20260927 02:47:39.004786 6172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:39.004786 6169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:39.005263 6164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:39.005479 6164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:39.006610 6164 hybrid_clock.cc:648] HybridClock initialized: now 1790477259006600 us; error 27 us; skew 500 ppm
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.009021 6164 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:47:39.009430 6164 webserver.cc:460] Webserver started at http://127.0.98.2:33491/ using document root <none> and password file <none>
I20260927 02:47:39.009620 6164 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:39.009667 6164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:39.009789 6164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:39.010874 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "d675a3cd04ea499ba45df23c6517ed8d"
format_stamp: "Formatted at 2026-09-27 02:47:39 on dist-test-slave-f4rg"
server_key: "a9d636a8c32460cb7ec5853b175b3ed8"
server_key_iv: "0839bd8e245bcac4594fec6c4029dd6d"
server_key_version: "encryptionkey@0"
I20260927 02:47:39.011206 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d675a3cd04ea499ba45df23c6517ed8d"
format_stamp: "Formatted at 2026-09-27 02:47:39 on dist-test-slave-f4rg"
server_key: "a9d636a8c32460cb7ec5853b175b3ed8"
server_key_iv: "0839bd8e245bcac4594fec6c4029dd6d"
server_key_version: "encryptionkey@0"
I20260927 02:47:39.012396 6164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:47:39.013120 6179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:39.013338 6164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:47:39.013422 6164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "d675a3cd04ea499ba45df23c6517ed8d"
format_stamp: "Formatted at 2026-09-27 02:47:39 on dist-test-slave-f4rg"
server_key: "a9d636a8c32460cb7ec5853b175b3ed8"
server_key_iv: "0839bd8e245bcac4594fec6c4029dd6d"
server_key_version: "encryptionkey@0"
I20260927 02:47:39.013506 6164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:39.020354 6164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:39.022390 6164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:39.022521 6164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:39.022727 6164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:39.023034 6164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:39.023067 6164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:39.023094 6164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:39.023119 6164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:39.028335 6164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39821
I20260927 02:47:39.028393 6292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39821 every 8 connection(s)
I20260927 02:47:39.028682 6164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:39.032070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6164
I20260927 02:47:39.032153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:39.032323 392 external_mini_cluster.cc:1428] Setting key 83fc1c82e90e4ae154efaf113d7114f2
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:39.037891 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:39.029296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57589 (local address 127.0.98.62:34541)
0927 02:47:39.029341 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.029343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.029559 (+ 216us) server_negotiation.cc:407] Connection header received
0927 02:47:39.030447 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.030448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.030458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.030519 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.031187 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.031671 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.032713 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.032928 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.032989 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:39.035138 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:39.035140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:39.035140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:39.035149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:39.036557 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.036969 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.036970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.036970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.037033 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.037245 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.037245 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.037246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.037381 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:39.037462 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:39.037710 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:47:39.037787 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:47:39.037974 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:39.029285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34541 (local address 127.0.98.2:57589)
0927 02:47:39.029438 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:39.029480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:39.030360 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:39.030601 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:39.030605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:39.030900 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:39.031118 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:39.031123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.031736 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.031736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:39.032639 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:39.032642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.033008 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.033009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:39.033129 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:39.033619 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:39.033628 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:39.035045 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:39.036664 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:39.036670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:39.036672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:39.036882 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:39.037097 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:39.037097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:39.037098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:39.037188 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:39.037452 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:39.037455 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:39.037587 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 02:47:39.037667 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:47:39.037784 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260927 02:47:39.038527 6293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34541
I20260927 02:47:39.038633 6293 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:39.038852 6293 heartbeater.cc:511] Master 127.0.98.62:34541 requested a full tablet report, sending...
I20260927 02:47:39.039243 5968 ts_manager.cc:194] Registered new tserver with Master: d675a3cd04ea499ba45df23c6517ed8d (127.0.98.2:39821)
I20260927 02:47:39.039631 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57589
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:47:39.057753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34541
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:39.136773 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:39.136976 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:39.136999 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:39.138142 6298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:39.138183 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:39.138211 6298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:39.139715 6298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:39.139932 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:39.140166 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:39.144987 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:39.144995 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:39.145040 6298 server_base.cc:1034] running on GCE node
W20260927 02:47:39.144995 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:39.145431 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:39.145641 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:39.146780 6298 hybrid_clock.cc:648] HybridClock initialized: now 1790477259146767 us; error 26 us; skew 500 ppm
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.153139 6298 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:47:39.153582 6298 webserver.cc:460] Webserver started at http://127.0.98.3:34135/ using document root <none> and password file <none>
I20260927 02:47:39.153764 6298 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:39.153829 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:39.153919 6298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:39.155000 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "238a1f7746d34d8fa41b167daa91abe2"
format_stamp: "Formatted at 2026-09-27 02:47:39 on dist-test-slave-f4rg"
server_key: "ca6a4ccdb4b11cac9e1f8d1edd3e2550"
server_key_iv: "2450cb94ea91536eb472783cecf90454"
server_key_version: "encryptionkey@0"
I20260927 02:47:39.155294 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "238a1f7746d34d8fa41b167daa91abe2"
format_stamp: "Formatted at 2026-09-27 02:47:39 on dist-test-slave-f4rg"
server_key: "ca6a4ccdb4b11cac9e1f8d1edd3e2550"
server_key_iv: "2450cb94ea91536eb472783cecf90454"
server_key_version: "encryptionkey@0"
I20260927 02:47:39.156486 6298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:39.157184 6313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:39.157384 6298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:39.157465 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "238a1f7746d34d8fa41b167daa91abe2"
format_stamp: "Formatted at 2026-09-27 02:47:39 on dist-test-slave-f4rg"
server_key: "ca6a4ccdb4b11cac9e1f8d1edd3e2550"
server_key_iv: "2450cb94ea91536eb472783cecf90454"
server_key_version: "encryptionkey@0"
I20260927 02:47:39.157518 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:39.172070 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:39.173925 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:39.174036 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:39.174219 6298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:39.174518 6298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:39.174552 6298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:39.174573 6298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:39.174592 6298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:39.179750 6298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42849
I20260927 02:47:39.179800 6426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42849 every 8 connection(s)
I20260927 02:47:39.180084 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:39.182364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6298
I20260927 02:47:39.182475 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:39.182670 392 external_mini_cluster.cc:1428] Setting key e04066e79e9b3686b435a734f7140f7a
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:47:39.189106 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:39.180824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53717 (local address 127.0.98.62:34541)
0927 02:47:39.180876 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.180878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.181097 (+ 219us) server_negotiation.cc:407] Connection header received
0927 02:47:39.182013 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.182014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.182025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.182087 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.182924 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.183555 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.184545 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.184740 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.184799 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:39.186801 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:39.186823 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:39.186823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:39.186831 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:39.188050 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.188399 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.188399 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.188400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.188445 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.188621 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.188622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.188622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.188712 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:39.188753 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:39.188989 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 02:47:39.189043 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:47:39.189291 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:39.180775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34541 (local address 127.0.98.3:53717)
0927 02:47:39.180941 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:39.180983 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:39.181901 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:39.182159 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:39.182162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:39.182526 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:39.182842 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:39.182850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.183637 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.183638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:39.184460 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:39.184462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.184815 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.184816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:39.184909 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:39.185414 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:39.185422 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:39.186727 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:39.188123 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:39.188129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:39.188130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:39.188338 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:39.188493 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:39.188493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:39.188493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:39.188580 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:39.188765 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:39.188768 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:39.188882 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 02:47:39.188959 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:47:39.189072 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260927 02:47:39.189720 6427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34541
I20260927 02:47:39.189815 6427 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:39.189978 6427 heartbeater.cc:511] Master 127.0.98.62:34541 requested a full tablet report, sending...
I20260927 02:47:39.190335 5968 ts_manager.cc:194] Registered new tserver with Master: 238a1f7746d34d8fa41b167daa91abe2 (127.0.98.3:42849)
I20260927 02:47:39.190696 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53717
I20260927 02:47:39.194381 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/test-admin.keytab.
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.208786 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 02:47:39.210289 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:39.209615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.0.98.1:37165)
0927 02:47:39.209786 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.209790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.209800 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:47:39.209825 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.209827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.209841 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.209911 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.210221 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47490: BlockingRecv error: recv got EOF from 127.0.0.1:47490 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
W20260927 02:47:39.210405 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.211359 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.212332 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:39.219966 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:39.212831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.98.1:37165)
0927 02:47:39.212890 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.212892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.212940 (+ 48us) server_negotiation.cc:407] Connection header received
0927 02:47:39.212991 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.212992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.213003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.213043 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.213486 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.214238 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.215013 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.215323 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.215373 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:39.217516 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:39.217524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:39.217525 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:39.217531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:39.218896 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.219212 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.219215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.219215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.219263 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.219425 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.219425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.219426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.219556 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:39.219710 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:39.219787 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:47:39.219862 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 02:47:39.221458 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:39.221080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.0.98.1:37165)
0927 02:47:39.221113 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.221114 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.221123 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:39.221161 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.221161 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.221170 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.221213 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.221419 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47514: BlockingRecv error: recv got EOF from 127.0.0.1:47514 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 02:47:39.221645 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.222499 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.223299 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:39.229869 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:39.223768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.0.98.1:37165)
0927 02:47:39.223817 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.223819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.223846 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:47:39.223903 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.223904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.223911 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.223952 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.224231 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.224708 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.225343 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.225528 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.225568 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:39.227473 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:39.227475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:39.227475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:39.227482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:39.228708 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.229111 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.229113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.229113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.229175 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.229402 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.229402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.229403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.229502 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:39.229587 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:39.229685 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:47:39.229762 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 02:47:39.231160 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:47:39.230689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.0.98.1:37165)
0927 02:47:39.230726 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.230728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.230776 (+ 48us) server_negotiation.cc:407] Connection header received
0927 02:47:39.230846 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.230847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.230855 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.230904 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.231116 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47524: BlockingRecv error: recv got EOF from 127.0.0.1:47524 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:47:39.231338 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790477232139228-392-0/krb5kdc/krb5cc)
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.232197 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:39.232903 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:47:39.239051 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:39.233365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.0.98.1:37165)
0927 02:47:39.233441 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:39.233443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:39.233450 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:47:39.233465 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:39.233465 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:39.233472 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:39.233513 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:39.233797 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.234248 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.234958 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:39.235138 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:39.235190 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:39.237046 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:39.237047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:39.237048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:39.237053 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:39.238226 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.238506 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.238506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.238507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.238552 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:39.238687 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:39.238688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:39.238688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:39.238760 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:39.238851 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:39.238939 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:47:39.238992 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 02:47:39.239578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6030
I20260927 02:47:39.244263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6164
I20260927 02:47:39.247991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6298
I20260927 02:47:39.251847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 5936
2026-09-27T02:47:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1707 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[6447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[6447](info): set up 2 sockets
Sep 27 02:47:39 dist-test-slave-f4rg krb5kdc[6447](info): commencing operation
krb5kdc: starting...
W20260927 02:47:40.202708 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:47:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:47:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:47:40.311659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:40.389863 6460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:40.390053 6460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:40.390072 6460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:40.391249 6460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:40.391305 6460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:40.391323 6460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:40.391340 6460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:40.391355 6460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:40.393082 6460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:40.393265 6460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:40.393440 6460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:40.397806 6466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:40.397814 6465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:40.397814 6468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:40.397889 6460 server_base.cc:1034] running on GCE node
I20260927 02:47:40.398252 6460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:40.398540 6460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:40.399672 6460 hybrid_clock.cc:648] HybridClock initialized: now 1790477260399660 us; error 28 us; skew 500 ppm
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:40.401961 6460 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:47:40.402415 6460 webserver.cc:460] Webserver started at http://127.0.98.62:38805/ using document root <none> and password file <none>
I20260927 02:47:40.402609 6460 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:40.402655 6460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:40.402781 6460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:40.403785 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "00bb59c7500f4d4584e9398ff6f9097b"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "05c05b8ab05e1b44158ce8c63f9776a1"
server_key_iv: "79947fa227281a0b30c558b5db5a32a5"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.404100 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "00bb59c7500f4d4584e9398ff6f9097b"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "05c05b8ab05e1b44158ce8c63f9776a1"
server_key_iv: "79947fa227281a0b30c558b5db5a32a5"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.405267 6460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:40.405893 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.406040 6460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:40.406116 6460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "00bb59c7500f4d4584e9398ff6f9097b"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "05c05b8ab05e1b44158ce8c63f9776a1"
server_key_iv: "79947fa227281a0b30c558b5db5a32a5"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.406175 6460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:40.423728 6460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:40.425841 6460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:40.425974 6460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:40.429653 6460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42157
I20260927 02:47:40.429683 6527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42157 every 8 connection(s)
I20260927 02:47:40.429999 6460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:47:40.430912 6528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:40.432996 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: Bootstrap starting.
I20260927 02:47:40.433486 6528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:40.433697 6528 log.cc:824] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:40.434304 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: No bootstrap required, opened a new log
I20260927 02:47:40.435802 6528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } }
I20260927 02:47:40.435950 6528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:40.435972 6528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00bb59c7500f4d4584e9398ff6f9097b, State: Initialized, Role: FOLLOWER
I20260927 02:47:40.436041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6460
I20260927 02:47:40.436095 6528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } }
I20260927 02:47:40.436143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:47:40.436165 6528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:47:40.436201 6528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:47:40.436252 6528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:40.436339 392 external_mini_cluster.cc:1428] Setting key 2fea71a09a74316e3fa6c2ec15bd5c8b
I20260927 02:47:40.437117 6528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } }
I20260927 02:47:40.437211 6528 leader_election.cc:302] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00bb59c7500f4d4584e9398ff6f9097b; no voters:
I20260927 02:47:40.437371 6528 leader_election.cc:288] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:47:40.437397 6533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:40.437522 6533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 1 LEADER]: Becoming Leader. State: Replica: 00bb59c7500f4d4584e9398ff6f9097b, State: Running, Role: LEADER
I20260927 02:47:40.437601 6528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:40.437608 6533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } }
I20260927 02:47:40.437958 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00bb59c7500f4d4584e9398ff6f9097b. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } }
I20260927 02:47:40.438042 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:40.437961 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } }
I20260927 02:47:40.438128 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:40.438225 6540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:47:40.438699 6540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:47:40.440129 6540 catalog_manager.cc:1347] Generated new cluster ID: 2183228d52164f0d95f897143d3c2a22
I20260927 02:47:40.440179 6540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:47:40.447273 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:40.436775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.0.98.62:42157)
0927 02:47:40.436939 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:40.436947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:40.436970 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:47:40.437060 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:40.437065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:40.437105 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:40.437224 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:40.438013 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.439476 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.440471 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.441058 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.441168 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:40.443651 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:40.443655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:40.443656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:40.443668 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:40.445635 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.445999 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.446002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.446003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.446139 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.446308 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.446309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.446309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.446623 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:40.446828 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:40.446902 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:47:40.447042 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260927 02:47:40.456465 6540 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:47:40.456960 6540 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:47:40.477108 6540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: Generated new TSK 0
I20260927 02:47:40.477298 6540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:47:40.499812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42157
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:40.578635 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:40.578862 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:40.578895 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:40.580030 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:40.580070 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:40.580103 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:40.581522 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:40.581755 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:40.581964 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:40.586946 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:40.586985 6554 server_base.cc:1034] running on GCE node
W20260927 02:47:40.586936 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:40.586968 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:40.587311 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:40.587522 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:40.588650 6554 hybrid_clock.cc:648] HybridClock initialized: now 1790477260588633 us; error 36 us; skew 500 ppm
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:40.591135 6554 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:47:40.591615 6554 webserver.cc:460] Webserver started at http://127.0.98.1:35561/ using document root <none> and password file <none>
I20260927 02:47:40.591830 6554 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:40.591881 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:40.592003 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:40.593012 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "2fbf107e71014a8e9e87c290dea96c46"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "d492daeeec55fcf6f925981f39ab63f0"
server_key_iv: "92da833010786aa1428cc46f902a0778"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.593346 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2fbf107e71014a8e9e87c290dea96c46"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "d492daeeec55fcf6f925981f39ab63f0"
server_key_iv: "92da833010786aa1428cc46f902a0778"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.594609 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:40.595389 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.595548 6554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.595628 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "2fbf107e71014a8e9e87c290dea96c46"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "d492daeeec55fcf6f925981f39ab63f0"
server_key_iv: "92da833010786aa1428cc46f902a0778"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.595696 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:40.605973 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:40.607712 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:40.607838 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:40.608036 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:40.608309 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:40.608340 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.608367 6554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:40.608390 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.613879 6554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43929
I20260927 02:47:40.613919 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43929 every 8 connection(s)
I20260927 02:47:40.614224 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:47:40.614516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6554
I20260927 02:47:40.614616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:47:40.614852 392 external_mini_cluster.cc:1428] Setting key feb8f0c4c67fd6dcd30fb235138149da
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:47:40.623287 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:40.614920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45557 (local address 127.0.98.62:42157)
0927 02:47:40.614970 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:40.614972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:40.615200 (+ 228us) server_negotiation.cc:407] Connection header received
0927 02:47:40.615799 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:40.615800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:40.615813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:40.615869 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:40.616859 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.617359 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.618354 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.618555 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.618610 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:40.620607 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:40.620609 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:40.620609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:40.620616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:40.622013 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.622435 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.622436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.622437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.622492 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.622663 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.622663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.622664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.622761 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:40.622853 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:40.623146 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 02:47:40.623219 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:47:40.623515 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:40.614925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:45557)
0927 02:47:40.615083 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:40.615131 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:40.615698 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:40.615931 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:40.615934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:40.616475 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:40.616747 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:40.616754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.617438 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.617439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:40.618272 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:40.618275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.618622 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.618623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:40.618747 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:40.619253 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:40.619262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:40.620528 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:40.622104 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:40.622110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:40.622112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:40.622367 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:40.622538 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:40.622539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:40.622539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:40.622623 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:40.622863 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:40.622870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:40.623033 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 02:47:40.623125 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:47:40.623290 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260927 02:47:40.623989 6683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42157
I20260927 02:47:40.624083 6683 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:40.624279 6683 heartbeater.cc:511] Master 127.0.98.62:42157 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 02:47:40.624835 6492 ts_manager.cc:194] Registered new tserver with Master: 2fbf107e71014a8e9e87c290dea96c46 (127.0.98.1:43929)
I20260927 02:47:40.625415 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:47:40.637979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42157
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:40.715816 6688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:40.716017 6688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:40.716040 6688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:40.717175 6688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:40.717218 6688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:40.717244 6688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:40.718659 6688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:40.718899 6688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:40.719113 6688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:40.723918 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:40.723901 6693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:40.723928 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:40.723934 6688 server_base.cc:1034] running on GCE node
I20260927 02:47:40.724359 6688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:40.724589 6688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:40.725745 6688 hybrid_clock.cc:648] HybridClock initialized: now 1790477260725729 us; error 34 us; skew 500 ppm
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:40.728235 6688 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:47:40.728698 6688 webserver.cc:460] Webserver started at http://127.0.98.2:38069/ using document root <none> and password file <none>
I20260927 02:47:40.728892 6688 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:40.728940 6688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:40.729063 6688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:40.730073 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "7b4c314b85564149bb9e5664bc05515a"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "1dc568bb7bc21fae18021e8ca7bc6dc7"
server_key_iv: "574c6dcd10ccf8d2835318475f867285"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.730403 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7b4c314b85564149bb9e5664bc05515a"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "1dc568bb7bc21fae18021e8ca7bc6dc7"
server_key_iv: "574c6dcd10ccf8d2835318475f867285"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.731647 6688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:47:40.732319 6703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.732467 6688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:40.732547 6688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "7b4c314b85564149bb9e5664bc05515a"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "1dc568bb7bc21fae18021e8ca7bc6dc7"
server_key_iv: "574c6dcd10ccf8d2835318475f867285"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.732609 6688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:40.741897 6688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:40.743681 6688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:40.743810 6688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:40.744020 6688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:40.744323 6688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:40.744356 6688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.744385 6688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:40.744402 6688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.749650 6688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36541
I20260927 02:47:40.749707 6816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36541 every 8 connection(s)
I20260927 02:47:40.749987 6688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:40.752291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6688
I20260927 02:47:40.752388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:47:40.752579 392 external_mini_cluster.cc:1428] Setting key 37ef429151e83584322834a68d9647ed
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:47:40.759727 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:40.750657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49401 (local address 127.0.98.62:42157)
0927 02:47:40.750731 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:40.750733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:40.750942 (+ 209us) server_negotiation.cc:407] Connection header received
0927 02:47:40.751423 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:40.751424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:40.751434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:40.751484 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:40.752417 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.753091 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.754428 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.754619 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.754682 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:40.756873 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:40.756874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:40.756874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:40.756883 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:40.758381 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.758742 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.758743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.758744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.758825 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.759056 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.759057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.759057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.759194 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:40.759262 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:40.759555 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 02:47:40.759630 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:47:40.759809 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:40.750629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:49401)
0927 02:47:40.750790 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:40.750859 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:40.751335 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:40.751541 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:40.751544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:40.752097 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:40.752287 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:40.752292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.753177 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.753178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:40.754346 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:40.754348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.754698 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.754700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:40.754873 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:40.755360 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:40.755367 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:40.756784 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:40.758464 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:40.758470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:40.758472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:40.758672 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:40.758887 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:40.758887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:40.758888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:40.758995 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:40.759305 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:40.759310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:40.759428 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 02:47:40.759513 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:47:40.759631 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260927 02:47:40.760277 6817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42157
I20260927 02:47:40.760370 6817 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:47:40.760542 6817 heartbeater.cc:511] Master 127.0.98.62:42157 requested a full tablet report, sending...
I20260927 02:47:40.760892 6492 ts_manager.cc:194] Registered new tserver with Master: 7b4c314b85564149bb9e5664bc05515a (127.0.98.2:36541)
I20260927 02:47:40.761221 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:47:40.775622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42157
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:40.853811 6822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:40.854025 6822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:40.854056 6822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:40.855209 6822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:40.855253 6822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:40.855288 6822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:40.856755 6822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:40.856998 6822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:40.857209 6822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:40.862193 6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:40.862244 6822 server_base.cc:1034] running on GCE node
W20260927 02:47:40.862185 6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:40.862187 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:40.862591 6822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:40.862826 6822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:40.864006 6822 hybrid_clock.cc:648] HybridClock initialized: now 1790477260863994 us; error 30 us; skew 500 ppm
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:40.866500 6822 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:47:40.866992 6822 webserver.cc:460] Webserver started at http://127.0.98.3:41993/ using document root <none> and password file <none>
I20260927 02:47:40.867187 6822 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:40.867236 6822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:40.867354 6822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:40.868422 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "5e9ff74b8fa9bcccce1c608051b85390"
server_key_iv: "8f559c20d9b03fdf1fbf16709406c8b9"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.868753 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "5e9ff74b8fa9bcccce1c608051b85390"
server_key_iv: "8f559c20d9b03fdf1fbf16709406c8b9"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.870090 6822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:40.870904 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.871059 6822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:40.871150 6822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "5e9ff74b8fa9bcccce1c608051b85390"
server_key_iv: "8f559c20d9b03fdf1fbf16709406c8b9"
server_key_version: "encryptionkey@0"
I20260927 02:47:40.871217 6822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:40.880864 6822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:40.882982 6822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:40.883111 6822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:40.883308 6822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:40.883594 6822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:40.883626 6822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.883653 6822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:40.883677 6822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:40.888945 6822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41703
I20260927 02:47:40.888995 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41703 every 8 connection(s)
I20260927 02:47:40.889307 6822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:47:40.889958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6822
I20260927 02:47:40.890056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:47:40.890233 392 external_mini_cluster.cc:1428] Setting key 74b5dd61a58396e6e4364aaa7b9279ba
Sep 27 02:47:40 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:47:40.897869 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:40.889976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41115 (local address 127.0.98.62:42157)
0927 02:47:40.890046 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:40.890049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:40.890233 (+ 184us) server_negotiation.cc:407] Connection header received
0927 02:47:40.890729 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:40.890730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:40.890740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:40.890786 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:40.891742 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.892228 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.893161 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.893340 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.893417 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:40.895563 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:40.895564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:40.895564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:40.895570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:40.896775 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.897109 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.897110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.897111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.897155 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:40.897328 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:40.897329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:40.897329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:40.897442 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:40.897487 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:40.897734 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:47:40.897793 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:47:40.898034 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:40.889916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:41115)
0927 02:47:40.890096 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:40.890139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:40.890656 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:40.890899 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:40.890903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:40.891464 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:40.891655 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:40.891660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.892300 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.892301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:40.893090 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:40.893092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:40.893447 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:40.893448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:40.893554 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:40.894038 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:40.894047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:40.895483 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:40.896844 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:40.896850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:40.896852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:40.897055 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:40.897196 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:40.897197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:40.897197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:40.897288 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:40.897509 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:40.897514 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:40.897627 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 02:47:40.897714 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:47:40.897846 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
I20260927 02:47:40.898500 6951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42157
I20260927 02:47:40.898612 6951 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:40.898778 6951 heartbeater.cc:511] Master 127.0.98.62:42157 requested a full tablet report, sending...
I20260927 02:47:40.899124 6492 ts_manager.cc:194] Registered new tserver with Master: 31f663fd4b464a1f9c1fcf19e24b8e8d (127.0.98.3:41703)
I20260927 02:47:40.899464 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41115
I20260927 02:47:40.901824 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 02:47:40.923418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46837
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42157,127.0.98.61:46837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:41.001372 6956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.001569 6956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.001588 6956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.002699 6956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:41.002746 6956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.002759 6956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.002771 6956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:41.002794 6956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:41.004284 6956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42157,127.0.98.61:46837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
W20260927 02:47:41.004384 6956 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42157,127.0.98.61:46837'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 02:47:41.004522 6956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.004714 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.009229 6961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.009227 6964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.009224 6962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.009244 6956 server_base.cc:1034] running on GCE node
I20260927 02:47:41.009624 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.009918 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.011053 6956 hybrid_clock.cc:648] HybridClock initialized: now 1790477261011038 us; error 27 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.013321 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:47:41.013763 6956 webserver.cc:460] Webserver started at http://127.0.98.61:40161/ using document root <none> and password file <none>
I20260927 02:47:41.013967 6956 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.014014 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.014137 6956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:41.015232 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/instance:
uuid: "504a1aee5b9c4329aec80a8ecf97109a"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "90e9d34fdb8e29a7cdf7b6e29ba656ca"
server_key_iv: "605958eb19e1fb51ad84b4fcea0f5651"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.015554 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal/instance:
uuid: "504a1aee5b9c4329aec80a8ecf97109a"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "90e9d34fdb8e29a7cdf7b6e29ba656ca"
server_key_iv: "605958eb19e1fb51ad84b4fcea0f5651"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.016772 6956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:47:41.017407 6971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.017551 6956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:41.017630 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
uuid: "504a1aee5b9c4329aec80a8ecf97109a"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "90e9d34fdb8e29a7cdf7b6e29ba656ca"
server_key_iv: "605958eb19e1fb51ad84b4fcea0f5651"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.017688 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.041108 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.042932 6956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.043066 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.046720 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46837
I20260927 02:47:41.046764 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46837 every 8 connection(s)
I20260927 02:47:41.047109 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:47:41.047693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 6956
I20260927 02:47:41.048192 7024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:41.050091 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:47:41.057114 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.048313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57112 (local address 127.0.98.61:46837)
0927 02:47:41.048469 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.048477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.048500 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:47:41.049168 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.049175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.049236 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.049356 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.049911 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.050972 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.051660 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.052066 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.052158 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.054365 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.054372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.054372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.054378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.055750 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.056062 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.056065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.056065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.056136 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.056300 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.056301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.056301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.056549 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.056668 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.056772 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.056895 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260927 02:47:41.057324 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.050668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41314)
0927 02:47:41.050836 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.050873 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.050937 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.051066 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.051069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.051366 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.051434 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.051436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.052079 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.052080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.052816 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.052818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.053136 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.053137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.053198 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.053599 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.053607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.054897 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.056282 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.056288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.056291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.056490 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.056658 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.056659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.056660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.056714 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.056982 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.056986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.057055 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.057139 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.057239 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260927 02:47:41.057338 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.050747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.0.98.62:42157)
0927 02:47:41.050819 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.050821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.050931 (+ 110us) server_negotiation.cc:407] Connection header received
0927 02:47:41.050996 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.050997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.051007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.051052 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.051511 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.051993 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.052889 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.053072 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.053124 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.054976 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.054977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.054978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.054985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.056202 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.056552 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.056554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.056554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.056607 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.056769 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.056770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.056770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.056910 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.056975 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.057172 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.057254 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:47:41.057834 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:47:41.063393 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.058062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.0.1:57118)
0927 02:47:41.058106 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.058116 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.058150 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.058231 (+ 81us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.058232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.058338 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.058380 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.058381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.058997 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.058998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.059568 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.059570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.059902 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.059903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.059947 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.060238 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.060241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.061383 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.062579 (+ 1196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.062580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.062580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.062730 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.062915 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.062916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.062916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.062973 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.063152 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.063153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.063184 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.063227 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.063301 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:47:41.063432 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.058088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.0.98.61:46837)
0927 02:47:41.058128 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.058130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.058140 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:47:41.058185 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.058186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.058192 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.058229 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.058417 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.058931 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.059652 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.059845 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.059905 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.061458 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.061459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.061460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.061465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.062508 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.062787 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.062787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.062787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.062854 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.063025 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.063025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.063025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.063111 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.063151 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.063246 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.063340 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":24}
I20260927 02:47:41.064668 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Bootstrap starting.
I20260927 02:47:41.065424 7024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:41.065658 7024 log.cc:824] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:41.066300 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: No bootstrap required, opened a new log
I20260927 02:47:41.067687 7024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } }
I20260927 02:47:41.067813 7024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:41.067854 7024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504a1aee5b9c4329aec80a8ecf97109a, State: Initialized, Role: FOLLOWER
I20260927 02:47:41.067966 7024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } }
I20260927 02:47:41.068178 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } } }
I20260927 02:47:41.068236 7024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:41.068262 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:47:41.069749 7041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:47:41.069806 7041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:47:41.071189 6956 master_runner.cc:428] Detected that this master 504a1aee5b9c4329aec80a8ecf97109a is joining an existing cluster
I20260927 02:47:41.071228 6956 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46837
I20260927 02:47:41.078972 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.072125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.0.98.62:42157)
0927 02:47:41.072167 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.072169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.072465 (+ 296us) server_negotiation.cc:407] Connection header received
0927 02:47:41.072492 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.072493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.072505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.072565 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.073088 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.073889 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.074723 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.075058 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.075130 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.076247 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.076249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.076249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.076256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.077760 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.078203 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.078204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.078204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.078265 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.078440 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.078441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.078441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.078556 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.078612 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.078782 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.078880 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:47:41.079387 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.072258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.0.98.61:46837)
0927 02:47:41.072306 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.072308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.072578 (+ 270us) server_negotiation.cc:407] Connection header received
0927 02:47:41.072596 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.072596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.072604 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.072638 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.073987 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.074704 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.075514 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.075779 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.075841 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.076903 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.076904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.076904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.076910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.078325 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.078667 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.078668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.078668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.078725 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.078922 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.078923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.078923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.079034 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.079094 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.079236 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.079292 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:47:41.079478 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.072096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.0.1:57130)
0927 02:47:41.072510 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.072516 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.072548 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.072660 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.072661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.072920 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.073016 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.073018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.074778 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.074779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.075446 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.075449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.075845 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.075846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.075900 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.076488 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.076493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.076830 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.078401 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.078402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.078402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.078614 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.078781 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.078782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.078782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.078860 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.079115 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.079115 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.079147 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.079324 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.079393 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260927 02:47:41.079310 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.072075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41322)
0927 02:47:41.072399 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.072407 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.072457 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.072669 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.072669 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.072915 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.073017 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.073018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.073982 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.073983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.074633 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.074636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.075146 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.075148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.075214 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.075781 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.075786 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.076175 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.077838 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.077838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.077839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.078128 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.078328 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.078328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.078329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.078390 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.078610 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.078612 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.078665 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.079092 (+ 427us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.079187 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:47:41.087507 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.081399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.98.61:46837)
0927 02:47:41.081430 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.081432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.081578 (+ 146us) server_negotiation.cc:407] Connection header received
0927 02:47:41.081660 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.081661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.081671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.081714 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.082089 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.082533 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.083305 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.083492 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.083544 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.085293 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.085294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.085294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.085300 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.086481 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.086827 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.086828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.086828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.086880 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.087036 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.087037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.087037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.087133 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.087184 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.087343 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.087412 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:47:41.087513 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.081382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.0.1:57134)
0927 02:47:41.081490 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.081524 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.081591 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.081717 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.081720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.081975 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.082020 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.082021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.082598 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.082599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.083237 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.083239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.083547 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.083548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.083631 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.084019 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.084024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.085217 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.086553 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.086560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.086562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.086750 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.086931 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.086932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.086932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.086989 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.087184 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.087192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.087242 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.087314 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.087419 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260927 02:47:41.088057 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46837: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46837
}
attrs {
promote: true
}
}
dest_uuid: "00bb59c7500f4d4584e9398ff6f9097b"
cas_config_opid_index: -1
I20260927 02:47:41.088390 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } }
I20260927 02:47:41.088824 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 504a1aee5b9c4329aec80a8ecf97109a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } }
I20260927 02:47:41.089094 6998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:41.089949 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46837
I20260927 02:47:41.090058 6998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Refusing update from remote peer 00bb59c7500f4d4584e9398ff6f9097b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 02:47:41.090093 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } }
I20260927 02:47:41.090160 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:41.090282 6956 master.cc:537] Master@127.0.98.61:46837 shutting down...
I20260927 02:47:41.091284 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Connected to new peer: Peer: permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:47:41.092360 6476 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 02:47:41.092829 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b -> Peer 504a1aee5b9c4329aec80a8ecf97109a (127.0.98.61:46837): Couldn't send request to peer 504a1aee5b9c4329aec80a8ecf97109a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:47:41.092872 6956 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:41.092959 6956 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:41.093003 6956 tablet_replica.cc:330] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: stopping tablet replica
I20260927 02:47:41.105540 6956 master.cc:552] Master@127.0.98.61:46837 shutdown complete.
I20260927 02:47:41.105587 6956 master_runner.cc:305] Clearing existing system tablet
I20260927 02:47:41.105700 6956 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.105739 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.106292 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260927 02:47:41.106678 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.106864 6956 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.106935 6956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.106961 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
uuid: "504a1aee5b9c4329aec80a8ecf97109a"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "90e9d34fdb8e29a7cdf7b6e29ba656ca"
server_key_iv: "605958eb19e1fb51ad84b4fcea0f5651"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.107713 6956 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:41.109340 6956 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:47:41.109408 6956 log.cc:1196] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 02:47:41.109596 6956 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Deleting consensus metadata
I20260927 02:47:41.109709 6956 master_runner.cc:315] Copying system tablet from 127.0.98.62:42157
I20260927 02:47:41.110298 6956 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42157
I20260927 02:47:41.116683 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.110537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41324)
0927 02:47:41.110648 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.110658 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.110717 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.110892 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.110893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.111204 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.111282 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.111284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.112238 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.112239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.112818 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.112825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.113251 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.113252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.113294 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.113750 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.113753 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.114037 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.115763 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.115764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.115765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.115923 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.116103 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.116103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.116104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.116157 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.116416 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.116417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.116458 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.116521 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.116603 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260927 02:47:41.116719 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.110612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.98.62:42157)
0927 02:47:41.110660 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.110662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.110704 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:47:41.110759 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.110760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.110770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.110884 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.111392 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.112168 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.112891 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.113192 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.113257 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.114113 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.114115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.114115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.114123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.115686 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.115984 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.115985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.115985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.116042 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.116208 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.116208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.116209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.116335 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.116399 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.116554 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.116626 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:47:41.117182 6512 tablet_copy_service.cc:140] P 00bb59c7500f4d4584e9398ff6f9097b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 504a1aee5b9c4329aec80a8ecf97109a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41324)
I20260927 02:47:41.117308 6512 tablet_copy_service.cc:161] P 00bb59c7500f4d4584e9398ff6f9097b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 504a1aee5b9c4329aec80a8ecf97109a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41324: session id = 504a1aee5b9c4329aec80a8ecf97109a-00000000000000000000000000000000
I20260927 02:47:41.117923 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:47:41.118508 6956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:47:41.119771 6956 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: tablet copy: Starting download of 0 data blocks...
I20260927 02:47:41.119923 6956 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: tablet copy: Starting download of 1 WAL segments...
I20260927 02:47:41.120702 6956 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:47:41.121568 6512 tablet_copy_service.cc:342] P 00bb59c7500f4d4584e9398ff6f9097b: Request end of tablet copy session 504a1aee5b9c4329aec80a8ecf97109a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41324
I20260927 02:47:41.121616 6512 tablet_copy_service.cc:433] P 00bb59c7500f4d4584e9398ff6f9097b: ending tablet copy session 504a1aee5b9c4329aec80a8ecf97109a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 504a1aee5b9c4329aec80a8ecf97109a
I20260927 02:47:41.125239 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.126366 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.126457 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.126629 6956 server_base.cc:1034] running on GCE node
W20260927 02:47:41.126914 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.127066 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.127189 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.128302 6956 hybrid_clock.cc:648] HybridClock initialized: now 1790477261128291 us; error 28 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.129487 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 02:47:41.129716 6956 webserver.cc:460] Webserver started at http://127.0.98.61:34923/ using document root <none> and password file <none>
I20260927 02:47:41.129827 6956 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.129861 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.130388 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:41.130879 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.130990 6956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.131031 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
uuid: "504a1aee5b9c4329aec80a8ecf97109a"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "90e9d34fdb8e29a7cdf7b6e29ba656ca"
server_key_iv: "605958eb19e1fb51ad84b4fcea0f5651"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.131088 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.147843 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.149322 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.151636 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46837
I20260927 02:47:41.151677 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46837 every 8 connection(s)
I20260927 02:47:41.151932 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:47:41.152577 7125 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:47:41.152945 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Bootstrap starting.
I20260927 02:47:41.155175 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:47:41.155370 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: Bootstrap complete.
I20260927 02:47:41.155546 7125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } }
I20260927 02:47:41.155610 7125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 504a1aee5b9c4329aec80a8ecf97109a, State: Initialized, Role: LEARNER
I20260927 02:47:41.155644 7125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } }
I20260927 02:47:41.155815 7125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:41.155810 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } }
I20260927 02:47:41.155905 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: This master's current role is: LEARNER
I20260927 02:47:41.157383 7139 catalog_manager.cc:1259] Loaded cluster ID: 2183228d52164f0d95f897143d3c2a22
I20260927 02:47:41.157424 7139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: loading cluster ID for follower catalog manager: success
I20260927 02:47:41.158160 7139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: acquiring CA information for follower catalog manager: success
I20260927 02:47:41.158459 7139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:47:41.165069 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.158570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.0.98.61:46837)
0927 02:47:41.158666 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.158667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.158701 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:47:41.158765 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.158766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.158776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.158873 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.159490 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.160092 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.161183 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.161371 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.161425 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.162735 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.162736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.162737 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.162743 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.164059 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.164432 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.164432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.164433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.164479 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.164662 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.164662 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.164662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.164756 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.164828 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.164947 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.165003 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":43,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 02:47:41.189307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35181
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:41.270959 7143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.271189 7143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.271216 7143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.272372 7143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:47:41.272411 7143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.272429 7143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.272444 7143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:47:41.272459 7143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:47:41.274004 7143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:41.274242 7143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.274484 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.279043 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.279052 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.279074 7143 server_base.cc:1034] running on GCE node
W20260927 02:47:41.279058 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.279439 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.279717 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.280857 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790477261280846 us; error 31 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.283164 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:47:41.283562 7143 webserver.cc:460] Webserver started at http://127.0.98.60:33785/ using document root <none> and password file <none>
I20260927 02:47:41.283756 7143 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.283798 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.283919 7143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:47:41.284936 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/instance:
uuid: "654d3e37bffc41a08e73999db16af9e2"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "caae0f149e0b6e96b75d299c7f0f430e"
server_key_iv: "30abbc70e08ede3184d9336704b104a2"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.285326 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal/instance:
uuid: "654d3e37bffc41a08e73999db16af9e2"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "caae0f149e0b6e96b75d299c7f0f430e"
server_key_iv: "30abbc70e08ede3184d9336704b104a2"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.286525 7143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:47:41.287235 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.287361 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.287436 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
uuid: "654d3e37bffc41a08e73999db16af9e2"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "caae0f149e0b6e96b75d299c7f0f430e"
server_key_iv: "30abbc70e08ede3184d9336704b104a2"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.287497 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.298379 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.300345 7143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.300457 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.304394 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35181
I20260927 02:47:41.304425 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35181 every 8 connection(s)
I20260927 02:47:41.304811 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:47:41.305706 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:41.307225 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:47:41.314230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 7143
I20260927 02:47:41.315753 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.308051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.0.98.62:42157)
0927 02:47:41.308100 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.308102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.308214 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:47:41.308687 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.308688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.308701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.308771 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.309417 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.309902 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.310898 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.311093 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.311144 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.313018 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.313019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.313020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.313026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.314349 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.314793 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.314794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.314795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.314879 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.315125 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.315127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.315127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.315267 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.315339 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.315560 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.315660 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:47:41.315864 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.307914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41336)
0927 02:47:41.308087 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.308135 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.308589 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.308859 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.308862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.309103 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.309331 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.309337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.309979 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.309980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.310784 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.310786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.311150 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.311151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.311247 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.311650 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.311657 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.312933 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.314425 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.314431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.314432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.314681 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.314936 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.314937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.314937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.315062 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.315329 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.315332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.315444 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.315521 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.315636 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260927 02:47:41.316352 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:47:41.322782 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.316628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57160 (local address 127.0.98.61:46837)
0927 02:47:41.316686 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.316687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.316695 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:41.316715 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.316716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.316729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.316774 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.317006 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.317476 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.318241 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.318455 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.318506 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.320396 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.320397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.320397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.320403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.321668 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.322025 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.322026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.322026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.322083 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.322254 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.322255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.322255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.322409 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.322477 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.322613 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.322692 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:47:41.322832 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.316563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.0.1:57160)
0927 02:47:41.316601 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.316611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.316643 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.316788 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.316788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.316903 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.316943 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.316945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.317558 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.317559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.318174 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.318176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.318510 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.318511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.318568 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.318904 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.318910 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.320312 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.321755 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.321756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.321756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.321958 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.322133 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.322134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.322134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.322204 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.322473 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.322474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.322516 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.322585 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.322678 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:47:41.322927 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.314738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.98.60:35181)
0927 02:47:41.314904 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.314909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.314929 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:47:41.314945 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.314947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.314959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.315029 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.315699 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.316606 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.317306 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.317611 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.317667 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.319987 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.319992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.319993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.319998 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.321599 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.322032 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.322035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.322036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.322082 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.322249 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.322250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.322250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.322405 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.322562 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.322607 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.322731 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260927 02:47:41.323167 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35181 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:47:41.328888 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.323381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.0.1:56220)
0927 02:47:41.323415 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.323424 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.323450 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.323648 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.323649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.323745 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.323780 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.323781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.324364 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.324365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.324849 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.324850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.325191 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.325191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.325234 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.325651 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.325655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.326756 (+ 1101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.328142 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.328143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.328143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.328297 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.328471 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.328472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.328472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.328518 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.328677 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.328677 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.328708 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.328751 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.328819 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:47:41.328948 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.323539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.98.60:35181)
0927 02:47:41.323572 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.323574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.323581 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:47:41.323596 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.323597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.323603 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.323644 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.323829 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.324284 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.324923 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.325114 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.325165 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.326869 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.326870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.326870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.326877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.328055 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.328385 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.328386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.328386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.328427 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.328556 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.328556 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.328557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.328638 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.328679 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.328768 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.328872 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":18}
I20260927 02:47:41.330202 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Bootstrap starting.
I20260927 02:47:41.331048 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:41.331314 7211 log.cc:824] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:41.331996 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: No bootstrap required, opened a new log
I20260927 02:47:41.333799 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35181 } }
I20260927 02:47:41.333940 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:41.333963 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654d3e37bffc41a08e73999db16af9e2, State: Initialized, Role: FOLLOWER
I20260927 02:47:41.334089 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35181 } }
I20260927 02:47:41.334383 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:41.334354 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35181 } } }
I20260927 02:47:41.334432 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:47:41.336058 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:47:41.336143 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:47:41.337990 7143 master_runner.cc:428] Detected that this master 654d3e37bffc41a08e73999db16af9e2 is joining an existing cluster
I20260927 02:47:41.338037 7143 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35181
I20260927 02:47:41.346460 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.339505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.98.61:46837)
0927 02:47:41.339551 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.339553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.339709 (+ 156us) server_negotiation.cc:407] Connection header received
0927 02:47:41.339796 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.339797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.339809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.339863 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.340246 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.341011 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342098 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.342398 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342458 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.343986 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.343987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.343988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.343994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.345390 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.345762 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.345762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.345763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.345810 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.346003 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.346005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.346005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.346125 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.346191 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.346327 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.346395 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:47:41.346529 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.339396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.0.1:57162)
0927 02:47:41.339649 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.339660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.339732 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.339867 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.339868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.340078 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.340159 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.340161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.341099 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.341100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.342008 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.342010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.342520 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.342579 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.343465 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.343470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.343902 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.345466 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.345467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.345468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.345691 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.345861 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.345861 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.345862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.345924 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.346191 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.346192 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.346237 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.346310 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.346417 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":117,"thread_start_us":77,"threads_started":1}
I20260927 02:47:41.362555 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.339520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.0.98.60:35181)
0927 02:47:41.339552 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.339553 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.339862 (+ 309us) server_negotiation.cc:407] Connection header received
0927 02:47:41.339937 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.339938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.339945 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.339967 (+ 22us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.340466 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.341168 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342198 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.342477 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342512 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.360190 (+ 17678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.360192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.360193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.360200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.361539 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.361842 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.361843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.361844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.361894 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.362034 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.362035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.362035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.362143 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.362194 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.362308 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.362427 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:47:41.362625 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.339620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.0.1:56226)
0927 02:47:41.339818 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.339825 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.339878 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.340012 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.340013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.340323 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.340392 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.340392 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.341268 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.341269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.342115 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.342116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.342578 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.342637 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.359572 (+ 16935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.359576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.360077 (+ 501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.361611 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.361612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.361612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.361785 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.361942 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.361942 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.361943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.361989 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.362190 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.362191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.362229 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.362286 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.362490 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":143,"thread_start_us":56,"threads_started":1}
I20260927 02:47:41.363132 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.339396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41338)
0927 02:47:41.339640 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.339650 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.339728 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.340013 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.340014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.340318 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.340390 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.340392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.341280 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.341280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.342115 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.342116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.342580 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342580 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.342636 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.359497 (+ 16861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.359503 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.360077 (+ 574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.362075 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.362077 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.362077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.362272 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.362482 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.362482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.362483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.362535 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.362801 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.362802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.362868 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.362932 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.363025 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260927 02:47:41.363145 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.339485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.0.98.62:42157)
0927 02:47:41.339526 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.339528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.339703 (+ 175us) server_negotiation.cc:407] Connection header received
0927 02:47:41.339918 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.339920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.339929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.339975 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.340474 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.341203 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342202 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.342485 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.342517 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.360479 (+ 17962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.360481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.360481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.360490 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.361978 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.362362 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.362363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.362363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.362416 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.362597 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.362598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.362598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.362721 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.362760 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.362979 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.363058 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477260, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:47:41.371311 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.365377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.0.98.60:35181)
0927 02:47:41.365419 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.365420 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.365427 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:47:41.365455 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.365456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.365464 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.365505 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.365745 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.366200 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.366900 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.367150 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.367204 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.369010 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.369012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.369012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.369018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.370203 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.370519 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.370520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.370521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.370572 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.370866 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.370867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.370867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.370967 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.371022 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.371154 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.371220 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:47:41.371312 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.365239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.0.1:56234)
0927 02:47:41.365280 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.365290 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.365324 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.365518 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.365518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.365653 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.365689 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.365690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.366264 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.366265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.366794 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.366796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.367210 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.367211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.367275 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.367666 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.367671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.368942 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.370275 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.370276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.370276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.370459 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.370622 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.370622 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.370623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.370676 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.371025 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.371026 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.371069 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.371128 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.371221 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:47:41.371907 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35181: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "654d3e37bffc41a08e73999db16af9e2"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35181
}
attrs {
promote: true
}
}
dest_uuid: "00bb59c7500f4d4584e9398ff6f9097b"
cas_config_opid_index: 5
I20260927 02:47:41.372129 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:41.372445 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 654d3e37bffc41a08e73999db16af9e2 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.372730 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:41.373440 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35181
I20260927 02:47:41.373517 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.373574 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:41.373577 7185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 1 FOLLOWER]: Refusing update from remote peer 00bb59c7500f4d4584e9398ff6f9097b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:47:41.373662 7143 master.cc:537] Master@127.0.98.60:35181 shutting down...
I20260927 02:47:41.374022 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Connected to new peer: Peer: permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 02:47:41.374907 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b -> Peer 654d3e37bffc41a08e73999db16af9e2 (127.0.98.60:35181): Couldn't send request to peer 654d3e37bffc41a08e73999db16af9e2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:47:41.376536 7143 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:41.376621 7143 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:41.376662 7143 tablet_replica.cc:330] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: stopping tablet replica
I20260927 02:47:41.378144 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.372490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.0.1:57166)
0927 02:47:41.373771 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.373782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.373825 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.375137 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.375138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.375140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:41.375174 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.375175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.375923 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.375923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.377141 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.377144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.377903 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.377904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.377969 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.377994 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.378040 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.378079 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1266}
I20260927 02:47:41.378540 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.372589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.0.98.61:46837)
0927 02:47:41.372702 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.372703 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.373874 (+ 1171us) server_negotiation.cc:407] Connection header received
0927 02:47:41.373901 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.373902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.373926 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.373978 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:41.375251 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.375842 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.377218 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.377821 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.377902 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.377977 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.378436 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.378476 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 02:47:41.378945 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 LEARNER]: Refusing update from remote peer 00bb59c7500f4d4584e9398ff6f9097b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 02:47:41.379151 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Connected to new peer: Peer: permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 02:47:41.381013 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 654d3e37bffc41a08e73999db16af9e2 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.382129 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.382208 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: This master's current role is: LEARNER
I20260927 02:47:41.382370 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00bb59c7500f4d4584e9398ff6f9097b. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: true } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.382437 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: This master's current role is: LEARNER
I20260927 02:47:41.382745 7050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: attempting to promote NON_VOTER 504a1aee5b9c4329aec80a8ecf97109a to VOTER
I20260927 02:47:41.382966 7050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:41.383337 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 LEARNER]: Refusing update from remote peer 00bb59c7500f4d4584e9398ff6f9097b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 02:47:41.383428 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.383319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.0.1:56246)
0927 02:47:41.383361 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.383400 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:47:41.383549 7238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [LEADER]: Connected to new peer: Peer: permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 02:47:41.383751 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.383639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.0.1:56256)
0927 02:47:41.383707 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.383732 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 02:47:41.383842 6477 consensus_peers.cc:487] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b -> Peer 654d3e37bffc41a08e73999db16af9e2 (127.0.98.60:35181): Couldn't send request to peer 654d3e37bffc41a08e73999db16af9e2. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 02:47:41.384079 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 504a1aee5b9c4329aec80a8ecf97109a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.384213 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 504a1aee5b9c4329aec80a8ecf97109a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.384766 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.384850 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:41.384827 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.384999 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00bb59c7500f4d4584e9398ff6f9097b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.385062 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:47:41.385169 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b [sys.catalog]: This master's current role is: LEADER
I20260927 02:47:41.393756 7143 master.cc:552] Master@127.0.98.60:35181 shutdown complete.
I20260927 02:47:41.393800 7143 master_runner.cc:305] Clearing existing system tablet
I20260927 02:47:41.393910 7143 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.393947 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.394457 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:47:41.394920 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.395051 7143 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.395102 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.395123 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
uuid: "654d3e37bffc41a08e73999db16af9e2"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "caae0f149e0b6e96b75d299c7f0f430e"
server_key_iv: "30abbc70e08ede3184d9336704b104a2"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.395697 7143 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:41.397152 7143 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 02:47:41.397207 7143 log.cc:1196] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 02:47:41.397369 7143 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Deleting consensus metadata
I20260927 02:47:41.397467 7143 master_runner.cc:315] Copying system tablet from 127.0.98.62:42157
I20260927 02:47:41.398021 7143 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42157
I20260927 02:47:41.403671 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.398372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.0.98.62:42157)
0927 02:47:41.398414 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.398416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.398528 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:47:41.398592 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.398593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.398603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.398670 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.399031 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.399564 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.400310 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.400508 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.400561 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.401401 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.401403 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.401403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.401410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.402663 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.403006 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.403007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.403007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.403062 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.403213 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.403214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.403214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.403320 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.403382 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.403511 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.403578 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:47:41.403681 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.398310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41350)
0927 02:47:41.398461 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.398470 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.398533 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.398693 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.398694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.398917 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.398958 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.398960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.399631 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.399632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.400236 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.400238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.400563 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.400564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.400623 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.401039 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.401044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.401324 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.402730 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.402731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.402731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.402947 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.403111 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.403111 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.403112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.403166 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.403378 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.403378 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.403422 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.403480 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.403593 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260927 02:47:41.404079 6512 tablet_copy_service.cc:140] P 00bb59c7500f4d4584e9398ff6f9097b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 654d3e37bffc41a08e73999db16af9e2 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41350)
I20260927 02:47:41.404155 6512 tablet_copy_service.cc:161] P 00bb59c7500f4d4584e9398ff6f9097b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 654d3e37bffc41a08e73999db16af9e2 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41350: session id = 654d3e37bffc41a08e73999db16af9e2-00000000000000000000000000000000
I20260927 02:47:41.404722 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:47:41.405145 7143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 02:47:41.406038 7143 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: tablet copy: Starting download of 0 data blocks...
I20260927 02:47:41.406181 7143 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: tablet copy: Starting download of 1 WAL segments...
I20260927 02:47:41.406870 7143 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:47:41.407594 6512 tablet_copy_service.cc:342] P 00bb59c7500f4d4584e9398ff6f9097b: Request end of tablet copy session 654d3e37bffc41a08e73999db16af9e2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41350
I20260927 02:47:41.407639 6512 tablet_copy_service.cc:433] P 00bb59c7500f4d4584e9398ff6f9097b: ending tablet copy session 654d3e37bffc41a08e73999db16af9e2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 654d3e37bffc41a08e73999db16af9e2
I20260927 02:47:41.409607 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.410622 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.410728 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.410734 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.410907 7143 server_base.cc:1034] running on GCE node
I20260927 02:47:41.410988 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.411113 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.412245 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790477261412227 us; error 27 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.413542 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 02:47:41.413801 7143 webserver.cc:460] Webserver started at http://127.0.98.60:42529/ using document root <none> and password file <none>
I20260927 02:47:41.413913 7143 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.413949 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.414616 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:47:41.415171 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.415272 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.415313 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
uuid: "654d3e37bffc41a08e73999db16af9e2"
format_stamp: "Formatted at 2026-09-27 02:47:41 on dist-test-slave-f4rg"
server_key: "caae0f149e0b6e96b75d299c7f0f430e"
server_key_iv: "30abbc70e08ede3184d9336704b104a2"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.415364 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.457345 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.458699 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.461619 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35181
I20260927 02:47:41.461661 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35181 every 8 connection(s)
I20260927 02:47:41.461930 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:47:41.462642 7310 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:47:41.463057 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Bootstrap starting.
I20260927 02:47:41.465341 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:47:41.465512 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: Bootstrap complete.
I20260927 02:47:41.465677 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:41.465747 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 654d3e37bffc41a08e73999db16af9e2, State: Initialized, Role: LEARNER
I20260927 02:47:41.465773 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:41.465935 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:47:41.465943 7313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } } }
I20260927 02:47:41.466068 7313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2 [sys.catalog]: This master's current role is: LEARNER
I20260927 02:47:41.467135 7324 catalog_manager.cc:1259] Loaded cluster ID: 2183228d52164f0d95f897143d3c2a22
I20260927 02:47:41.467213 7324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: loading cluster ID for follower catalog manager: success
I20260927 02:47:41.468475 7324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: acquiring CA information for follower catalog manager: success
I20260927 02:47:41.468772 7324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 654d3e37bffc41a08e73999db16af9e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:47:41.489962 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.483725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56272 (local address 127.0.98.60:35181)
0927 02:47:41.483891 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.483893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.483904 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:47:41.483928 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.483929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.483938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.484016 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.484545 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.485190 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.485919 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.486135 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.486200 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.487586 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.487588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.487588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.487596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.488989 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.489362 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.489362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.489362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.489415 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.489547 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.489548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.489548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.489643 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.489722 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.489822 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.489884 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260927 02:47:41.490446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6554
I20260927 02:47:41.496589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43929
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42157
--webserver_port=35561
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:47:41.576879 7326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.577093 7326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.577116 7326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.578279 7326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.578321 7326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.578346 7326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:47:41.579865 7326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35561
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:41.580097 7326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.580334 7326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.585129 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.585114 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.585201 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.585232 7326 server_base.cc:1034] running on GCE node
I20260927 02:47:41.585500 7326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.585716 7326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.586871 7326 hybrid_clock.cc:648] HybridClock initialized: now 1790477261586865 us; error 34 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.589349 7326 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:47:41.589782 7326 webserver.cc:460] Webserver started at http://127.0.98.1:35561/ using document root <none> and password file <none>
I20260927 02:47:41.589998 7326 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.590062 7326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.591467 7326 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:41.592059 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.592236 7326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:41.592317 7326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "2fbf107e71014a8e9e87c290dea96c46"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "d492daeeec55fcf6f925981f39ab63f0"
server_key_iv: "92da833010786aa1428cc46f902a0778"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.592653 7326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.605160 7326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.607441 7326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.607561 7326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.607789 7326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:41.608085 7326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:41.608117 7326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.608139 7326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:41.608161 7326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.613493 7326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43929
I20260927 02:47:41.613543 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43929 every 8 connection(s)
I20260927 02:47:41.613880 7326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 02:47:41.621130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 7326
I20260927 02:47:41.621291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6688
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:47:41.625583 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.614573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50155 (local address 127.0.98.61:46837)
0927 02:47:41.614883 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.614885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.614973 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:47:41.615551 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.615553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.615564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.615593 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.616545 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.617284 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.618606 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.618842 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.618901 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.622761 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.622763 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.622763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.622770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.624200 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624652 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.624653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.624653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.624709 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624952 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.624953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.624953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.625088 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.625125 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.625407 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.625487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260927 02:47:41.625831 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.614582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.98.1:50155)
0927 02:47:41.614844 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.614889 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.615437 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.615665 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.615669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.616164 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.616422 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.616426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.617396 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.617397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.618520 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.618522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.618912 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.618913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.619048 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.619546 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.619555 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.622674 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.624285 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.624286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.624572 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.624781 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.624782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.624891 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.625163 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.625163 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.625272 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.625367 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.625550 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260927 02:47:41.625835 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.614582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:48053)
0927 02:47:41.614961 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.614972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.615442 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.615951 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.615952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.616182 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.616420 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.616424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.618611 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.618611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.619360 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.619362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.619799 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.619800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.619862 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.620200 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.620204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.622674 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.624255 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.624265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.624563 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.624765 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.624766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.624888 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.625110 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.625116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.625270 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.625368 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.625617 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":119,"thread_start_us":86,"threads_started":1}
I20260927 02:47:41.625907 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.614924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48053 (local address 127.0.98.62:42157)
0927 02:47:41.614991 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.614993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.615019 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:47:41.615519 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.615520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.615532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.615577 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.616500 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.617275 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.619431 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.619725 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.619785 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.622890 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.622892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.622892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.622899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.624183 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624645 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.624646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.624646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.624711 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.624942 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.624943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.624944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.625059 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.625112 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.625737 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.625825 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:47:41.626315 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46837
I20260927 02:47:41.626384 7457 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.626458 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42157
I20260927 02:47:41.626502 7456 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.626518 7457 heartbeater.cc:511] Master 127.0.98.61:46837 requested a full tablet report, sending...
I20260927 02:47:41.626559 7456 heartbeater.cc:511] Master 127.0.98.62:42157 requested a full tablet report, sending...
I20260927 02:47:41.627019 7089 ts_manager.cc:194] Registered new tserver with Master: 2fbf107e71014a8e9e87c290dea96c46 (127.0.98.1:43929)
I20260927 02:47:41.627207 6492 ts_manager.cc:194] Re-registered known tserver with Master: 2fbf107e71014a8e9e87c290dea96c46 (127.0.98.1:43929)
I20260927 02:47:41.627583 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48053
I20260927 02:47:41.628260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36541
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42157
--webserver_port=38069
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:47:41.629702 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.614582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.98.1:54995)
0927 02:47:41.615038 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.615046 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.615453 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.615669 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.615670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.616192 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.616420 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.616426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.617909 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.617910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.618684 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.618685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.619096 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.619097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.619160 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.619680 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.619684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.621270 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.628789 (+ 7519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.628790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.628791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.628958 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.629126 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.629127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.629127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.629192 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.629439 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.629440 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.629480 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.629539 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.629621 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":50,"thread_start_us":82,"threads_started":1}
I20260927 02:47:41.629742 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.614705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54995 (local address 127.0.98.60:35181)
0927 02:47:41.614748 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.614750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.615093 (+ 343us) server_negotiation.cc:407] Connection header received
0927 02:47:41.615554 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.615555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.615568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.615595 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.617372 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.617828 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.618771 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.619035 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.619086 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.621380 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.621381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.621382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.621389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.628717 (+ 7328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.629019 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.629020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.629021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.629075 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.629243 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.629243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.629244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.629372 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.629435 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.629570 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.629654 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:47:41.630052 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35181
I20260927 02:47:41.630102 7455 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.630177 7455 heartbeater.cc:511] Master 127.0.98.60:35181 requested a full tablet report, sending...
I20260927 02:47:41.630503 7274 ts_manager.cc:194] Registered new tserver with Master: 2fbf107e71014a8e9e87c290dea96c46 (127.0.98.1:43929)
W20260927 02:47:41.708163 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.708407 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.708441 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.709595 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.709640 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.709676 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:47:41.711186 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38069
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:41.711418 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.711630 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.716352 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.716387 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.716424 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.716491 7462 server_base.cc:1034] running on GCE node
I20260927 02:47:41.716747 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.716989 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.718122 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790477261718103 us; error 34 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.720583 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:47:41.721001 7462 webserver.cc:460] Webserver started at http://127.0.98.2:38069/ using document root <none> and password file <none>
I20260927 02:47:41.721200 7462 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.721264 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.722599 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:41.723207 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.723392 7462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.723475 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "7b4c314b85564149bb9e5664bc05515a"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "1dc568bb7bc21fae18021e8ca7bc6dc7"
server_key_iv: "574c6dcd10ccf8d2835318475f867285"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.723834 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.743474 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.745430 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.745537 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.745712 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:41.745965 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:41.745993 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.746018 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:41.746034 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.751477 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36541
I20260927 02:47:41.751538 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36541 every 8 connection(s)
I20260927 02:47:41.751859 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:47:41.753094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 7462
I20260927 02:47:41.753217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6822
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:47:41.759603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41703
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42157
--webserver_port=41993
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:47:41.763259 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.752513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.98.2:42207)
0927 02:47:41.752699 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.752739 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.753239 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.753555 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.753559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.754124 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.754324 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.754329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.755290 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.755291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.756612 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.756614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.757190 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.757191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.757331 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.757866 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.757874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.759675 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.761580 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.761588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.761590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.761857 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.762086 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762086 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.762087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.762223 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.762521 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.762527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.762697 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.762843 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.763016 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260927 02:47:41.763293 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.752853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50649 (local address 127.0.98.62:42157)
0927 02:47:41.752903 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.752905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.754542 (+ 1637us) server_negotiation.cc:407] Connection header received
0927 02:47:41.754573 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.754574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.754586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.754635 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.755199 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.755966 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.756987 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.757271 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.757309 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.760726 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.760728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.760728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.760735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.762118 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762530 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.762531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.762531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.762587 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762763 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.762764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.762764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.762918 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.762958 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.763140 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.763214 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:47:41.763466 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.752795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34671 (local address 127.0.98.60:35181)
0927 02:47:41.752857 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.752859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.754618 (+ 1759us) server_negotiation.cc:407] Connection header received
0927 02:47:41.754707 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.754709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.754719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.754744 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.755570 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.756288 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.757200 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.757479 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.757518 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.760692 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.760693 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.760693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.760701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.762075 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762509 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.762510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.762510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.762565 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762727 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.762727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.762728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.762878 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.762935 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.763314 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.763388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:47:41.763557 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.752513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:50649)
0927 02:47:41.754443 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.754452 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.754499 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.754684 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.754685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.755031 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.755104 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.755107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.756085 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.756086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.756897 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.756900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.757393 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.757394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.757455 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.758229 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.758234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.760634 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.762189 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.762190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.762453 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.762648 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.762649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.762700 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.763002 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.763003 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.763040 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.763355 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.763454 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1902,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260927 02:47:41.763265 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.752513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.98.2:34671)
0927 02:47:41.754571 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.754578 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.754642 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.754825 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.754826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.755434 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.755485 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.755486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.756376 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.756377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.757121 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.757123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.757588 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.757588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.757632 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.757979 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.757983 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.760563 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.762149 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.762150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.762439 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.762621 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.762622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.762672 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.762960 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.762961 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.763005 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.763060 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.763150 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"mutex_wait_us":25,"thread_start_us":82,"threads_started":1}
I20260927 02:47:41.764007 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42157
I20260927 02:47:41.764106 7592 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.764294 7592 heartbeater.cc:511] Master 127.0.98.62:42157 requested a full tablet report, sending...
I20260927 02:47:41.764451 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35181
I20260927 02:47:41.764654 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.752795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42207 (local address 127.0.98.61:46837)
0927 02:47:41.752898 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.752900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.752910 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:47:41.753359 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.753360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.753372 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.753396 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.754414 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.755181 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.756727 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.757049 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.757087 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.759782 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.759783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.759784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.759792 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.761497 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.761936 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.761938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.761938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.761999 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.762300 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.762301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.762301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.762439 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.762498 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.764475 (+ 1977us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.764553 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 02:47:41.764703 6492 ts_manager.cc:194] Re-registered known tserver with Master: 7b4c314b85564149bb9e5664bc05515a (127.0.98.2:36541)
I20260927 02:47:41.765197 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50649
I20260927 02:47:41.765149 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46837
I20260927 02:47:41.765439 7591 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.765494 7593 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.765604 7591 heartbeater.cc:511] Master 127.0.98.60:35181 requested a full tablet report, sending...
I20260927 02:47:41.765609 7593 heartbeater.cc:511] Master 127.0.98.61:46837 requested a full tablet report, sending...
I20260927 02:47:41.765949 7274 ts_manager.cc:194] Registered new tserver with Master: 7b4c314b85564149bb9e5664bc05515a (127.0.98.2:36541)
I20260927 02:47:41.765947 7089 ts_manager.cc:194] Registered new tserver with Master: 7b4c314b85564149bb9e5664bc05515a (127.0.98.2:36541)
W20260927 02:47:41.841270 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:47:41.841475 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:47:41.841498 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:47:41.842675 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:47:41.842720 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:47:41.842746 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:47:41.844198 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41993
--tserver_master_addrs=127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:47:41.844419 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:47:41.844636 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:47:41.849375 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.849385 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:47:41.849853 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:47:41.849905 7598 server_base.cc:1034] running on GCE node
I20260927 02:47:41.850114 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:47:41.850349 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:47:41.851488 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790477261851474 us; error 28 us; skew 500 ppm
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:47:41.853813 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:47:41.854242 7598 webserver.cc:460] Webserver started at http://127.0.98.3:41993/ using document root <none> and password file <none>
I20260927 02:47:41.854440 7598 fs_manager.cc:359] Metadata directory not provided
I20260927 02:47:41.854501 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:47:41.855863 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:41.856514 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.856650 7598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:47:41.856725 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d"
format_stamp: "Formatted at 2026-09-27 02:47:40 on dist-test-slave-f4rg"
server_key: "5e9ff74b8fa9bcccce1c608051b85390"
server_key_iv: "8f559c20d9b03fdf1fbf16709406c8b9"
server_key_version: "encryptionkey@0"
I20260927 02:47:41.857079 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:47:41.868009 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:47:41.869904 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:47:41.870023 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:47:41.870249 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:47:41.870538 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:47:41.870571 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.870600 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:47:41.870617 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:47:41.876024 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41703
I20260927 02:47:41.876062 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41703 every 8 connection(s)
I20260927 02:47:41.876382 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 02:47:41 dist-test-slave-f4rg krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477261, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 02:47:41.886066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 7598
I20260927 02:47:41.887602 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.877134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34821 (local address 127.0.98.60:35181)
0927 02:47:41.877197 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.877199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.877918 (+ 719us) server_negotiation.cc:407] Connection header received
0927 02:47:41.878411 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.878412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.878425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.878480 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.879365 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.880142 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881498 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.881783 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881836 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.884504 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.884506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.884506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.884513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.886080 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.886535 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.886537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.886537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.886596 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.886897 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.886898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.886898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.887033 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.887076 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.887435 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.887516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:47:41.888127 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.877034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35181 (local address 127.0.98.3:34821)
0927 02:47:41.877795 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.877836 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.878322 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.878600 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.878605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.879031 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.879283 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.879288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.880239 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.880240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.881426 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.881428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.881879 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.881933 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.882568 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.882578 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.884414 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.886167 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.886174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.886177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.886447 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.886667 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.886668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.886668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.886823 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.887123 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.887129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.887288 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.887732 (+ 444us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.887909 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":26,"thread_start_us":661,"threads_started":1}
I20260927 02:47:41.888387 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.877034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:56209)
0927 02:47:41.878871 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.878880 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.878935 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.879066 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.879066 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.879197 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.879284 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.879289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.880239 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.880240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.881475 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.881477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.881875 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.881925 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.882656 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.882659 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.884865 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.887353 (+ 2488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.887354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.887355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.887575 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.887778 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.887779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.887779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.887831 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.888102 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.888103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.888141 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.888198 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.888287 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1543,"mutex_wait_us":223,"thread_start_us":528,"threads_started":1}
I20260927 02:47:41.888522 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.877034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46837 (local address 127.0.98.3:47095)
0927 02:47:41.878475 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.878484 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.878537 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.878702 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.878703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.879030 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:47:41.879284 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.879289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.880756 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.880757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.881363 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.881365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.881792 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.881922 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.882585 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:47:41.882588 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:47:41.885970 (+ 3382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:47:41.887575 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.887575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.887576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.887787 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.887969 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:47:41.887969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:47:41.887970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:47:41.888017 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:47:41.888276 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:47:41.888277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:47:41.888310 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.888368 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.888440 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1415,"mutex_wait_us":3,"thread_start_us":665,"threads_started":1}
I20260927 02:47:41.888641 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.877261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56209 (local address 127.0.98.62:42157)
0927 02:47:41.877316 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.877318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.878939 (+ 1621us) server_negotiation.cc:407] Connection header received
0927 02:47:41.878988 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.878989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.879001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.879049 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.879398 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.880130 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881568 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.881758 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881824 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.884942 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.884944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.884944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.884952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.887277 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.887647 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.887648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.887649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.887704 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.887892 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.887893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.887893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.888025 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.888079 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.888491 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.888561 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:47:41.888715 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35181
I20260927 02:47:41.888806 7727 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.888813 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.877171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47095 (local address 127.0.98.61:46837)
0927 02:47:41.877244 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.877247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.878557 (+ 1310us) server_negotiation.cc:407] Connection header received
0927 02:47:41.878584 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.878586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.878598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.878681 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.880235 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.880684 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881439 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.881717 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.881780 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.886068 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.886069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.886070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.886076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.887490 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.887856 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.887857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.887858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.887907 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.888076 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.888077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.888077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.888191 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.888242 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.888686 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.888756 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:47:41.888981 7727 heartbeater.cc:511] Master 127.0.98.60:35181 requested a full tablet report, sending...
I20260927 02:47:41.889066 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42157
I20260927 02:47:41.889108 7728 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.889129 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46837
I20260927 02:47:41.889160 7729 heartbeater.cc:464] Registering TS with master...
I20260927 02:47:41.889163 7728 heartbeater.cc:511] Master 127.0.98.62:42157 requested a full tablet report, sending...
I20260927 02:47:41.889236 7729 heartbeater.cc:511] Master 127.0.98.61:46837 requested a full tablet report, sending...
I20260927 02:47:41.889464 6492 ts_manager.cc:194] Re-registered known tserver with Master: 31f663fd4b464a1f9c1fcf19e24b8e8d (127.0.98.3:41703)
I20260927 02:47:41.889545 7089 ts_manager.cc:194] Registered new tserver with Master: 31f663fd4b464a1f9c1fcf19e24b8e8d (127.0.98.3:41703)
I20260927 02:47:41.889541 7274 ts_manager.cc:194] Registered new tserver with Master: 31f663fd4b464a1f9c1fcf19e24b8e8d (127.0.98.3:41703)
I20260927 02:47:41.889883 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56209
I20260927 02:47:41.896673 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.888073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56276 (local address 127.0.98.60:35181)
0927 02:47:41.888108 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.888109 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.890073 (+ 1964us) server_negotiation.cc:407] Connection header received
0927 02:47:41.890141 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.890142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.890150 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.890196 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.890908 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.891417 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.892144 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.892425 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.892489 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.894079 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.894080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.894080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.894086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.895495 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.895851 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.895851 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.895852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.895902 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.896066 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.896067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.896067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.896211 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.896293 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.896511 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.896582 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:47:41.896834 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.888283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.0.98.62:42157)
0927 02:47:41.888369 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.888370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.889953 (+ 1583us) server_negotiation.cc:407] Connection header received
0927 02:47:41.890001 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.890002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.890014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.890081 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.890883 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.891555 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.892517 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.892761 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.892798 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.894176 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.894177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.894178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.894184 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.895787 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.896147 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.896148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.896149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.896202 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.896370 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.896370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.896371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.896487 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.896574 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.896666 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.896743 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":43,"threads_started":1}
I20260927 02:47:41.897015 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.887638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.98.61:46837)
0927 02:47:41.888869 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.888871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.890582 (+ 1711us) server_negotiation.cc:407] Connection header received
0927 02:47:41.890620 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.890621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.890630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.890672 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.891324 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.891968 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.892963 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.893211 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.893269 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.894499 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.894500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.894500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.894508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.895934 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.896251 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.896252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.896253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.896304 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.896488 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.896489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.896489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.896604 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.896679 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.896861 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.896931 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":49,"threads_started":1}
I20260927 02:47:41.898430 6492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:47:41.899971 6492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:47:41.906481 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.903181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36541 (local address 127.0.0.1:34258)
0927 02:47:41.903423 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.903432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.903480 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.903702 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.903703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.903705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:41.903757 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.903759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.904640 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.904641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.905360 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.905362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.906258 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.906259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.906313 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.906317 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.906374 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.906412 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":53,"threads_started":1}
I20260927 02:47:41.906502 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.903381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.0.98.2:36541)
0927 02:47:41.903571 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.903575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.903584 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:47:41.903615 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.903617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.903628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.903693 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:41.903821 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.904567 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.905439 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.906164 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.906253 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.906324 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.906387 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.906425 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260927 02:47:41.907256 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.903076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41703 (local address 127.0.0.1:44878)
0927 02:47:41.903412 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.903421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.903473 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.903674 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.903675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.903677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:41.903735 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.903737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.905332 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.905333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.906199 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.906201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.907044 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.907045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.907103 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.907106 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.907157 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.907191 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":28,"thread_start_us":80,"threads_started":1}
I20260927 02:47:41.907289 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.903278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.98.3:41703)
0927 02:47:41.903409 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.903414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.903466 (+ 52us) server_negotiation.cc:407] Connection header received
0927 02:47:41.903541 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.903543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.903561 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.903811 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:41.903871 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.905262 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.906285 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.906974 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.907035 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.907105 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.907171 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.907208 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260927 02:47:41.907356 7525 tablet_service.cc:1467] Processing CreateTablet for tablet f9bfcf9d5c8f4979804706f1953e36b8 (DEFAULT_TABLE table=test-table [id=72f6365181814425b2e9172a887bd61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:41.907732 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bfcf9d5c8f4979804706f1953e36b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:41.907795 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.903028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43929 (local address 127.0.0.1:47468)
0927 02:47:41.903197 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.903208 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.903254 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.903944 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.903944 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.903947 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:41.904011 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.904013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.905302 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.905303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.906451 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.906454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.907572 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.907573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.907631 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.907635 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.907663 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.907706 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":37,"thread_start_us":72,"threads_started":1}
I20260927 02:47:41.907825 7661 tablet_service.cc:1467] Processing CreateTablet for tablet f9bfcf9d5c8f4979804706f1953e36b8 (DEFAULT_TABLE table=test-table [id=72f6365181814425b2e9172a887bd61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:41.908136 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bfcf9d5c8f4979804706f1953e36b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:41.908170 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.903122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.0.98.1:43929)
0927 02:47:41.903585 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.903590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.903598 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:41.903631 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.903633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.903647 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.904077 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:41.904142 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.905213 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.906650 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.907475 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.907936 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.908038 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.908062 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.908097 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":120,"threads_started":1}
I20260927 02:47:41.908803 7389 tablet_service.cc:1467] Processing CreateTablet for tablet f9bfcf9d5c8f4979804706f1953e36b8 (DEFAULT_TABLE table=test-table [id=72f6365181814425b2e9172a887bd61b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:47:41.909103 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9bfcf9d5c8f4979804706f1953e36b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:47:41.911042 7754 tablet_bootstrap.cc:492] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Bootstrap starting.
I20260927 02:47:41.911669 7754 tablet_bootstrap.cc:654] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:41.911648 7755 tablet_bootstrap.cc:492] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Bootstrap starting.
I20260927 02:47:41.912467 7754 log.cc:824] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:41.912730 7755 tablet_bootstrap.cc:654] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:41.913012 7755 log.cc:824] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:41.913213 7754 tablet_bootstrap.cc:492] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: No bootstrap required, opened a new log
I20260927 02:47:41.913285 7754 ts_tablet_manager.cc:1397] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:47:41.913303 7756 tablet_bootstrap.cc:492] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Bootstrap starting.
I20260927 02:47:41.913805 7755 tablet_bootstrap.cc:492] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: No bootstrap required, opened a new log
I20260927 02:47:41.913873 7756 tablet_bootstrap.cc:654] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Neither blocks nor log segments found. Creating new log.
I20260927 02:47:41.913921 7755 ts_tablet_manager.cc:1397] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:41.914145 7756 log.cc:824] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Log is configured to *not* fsync() on all Append() calls
I20260927 02:47:41.914891 7756 tablet_bootstrap.cc:492] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: No bootstrap required, opened a new log
I20260927 02:47:41.914956 7756 ts_tablet_manager.cc:1397] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:47:41.914970 7754 raft_consensus.cc:348] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.915146 7754 raft_consensus.cc:374] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:41.915185 7754 raft_consensus.cc:729] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f663fd4b464a1f9c1fcf19e24b8e8d, State: Initialized, Role: FOLLOWER
I20260927 02:47:41.915292 7754 consensus_queue.cc:260] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.915503 7754 ts_tablet_manager.cc:1428] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:47:41.915717 7728 heartbeater.cc:503] Master 127.0.98.62:42157 was elected leader, sending a full tablet report...
I20260927 02:47:41.917232 7755 raft_consensus.cc:348] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.917393 7755 raft_consensus.cc:374] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:41.917428 7755 raft_consensus.cc:729] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4c314b85564149bb9e5664bc05515a, State: Initialized, Role: FOLLOWER
I20260927 02:47:41.917450 7756 raft_consensus.cc:348] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.917546 7755 consensus_queue.cc:260] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.917586 7756 raft_consensus.cc:374] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:47:41.917626 7756 raft_consensus.cc:729] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fbf107e71014a8e9e87c290dea96c46, State: Initialized, Role: FOLLOWER
I20260927 02:47:41.917752 7756 consensus_queue.cc:260] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.917790 7755 ts_tablet_manager.cc:1428] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 02:47:41.917806 7592 heartbeater.cc:503] Master 127.0.98.62:42157 was elected leader, sending a full tablet report...
I20260927 02:47:41.918416 7756 ts_tablet_manager.cc:1428] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:47:41.918721 7456 heartbeater.cc:503] Master 127.0.98.62:42157 was elected leader, sending a full tablet report...
I20260927 02:47:41.922961 7762 raft_consensus.cc:482] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:47:41.923041 7762 raft_consensus.cc:504] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.923355 7762 leader_election.cc:288] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31f663fd4b464a1f9c1fcf19e24b8e8d (127.0.98.3:41703), 7b4c314b85564149bb9e5664bc05515a (127.0.98.2:36541)
I20260927 02:47:41.926734 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.923514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36541 (local address 127.0.98.1:37985)
0927 02:47:41.923553 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.923560 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.923586 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.923720 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.923721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.923723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:41.923811 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.923813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.924660 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.924660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.925932 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.925935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.926464 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.926465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.926541 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.926545 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.926601 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.926644 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":38}
I20260927 02:47:41.926775 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.923540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37985 (local address 127.0.98.2:36541)
0927 02:47:41.923623 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.923625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.923633 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:41.923655 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.923655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.923666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.923718 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:41.923860 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.924591 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.926012 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.926391 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.926442 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.926454 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.926608 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.926676 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 02:47:41.927513 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.923582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55933 (local address 127.0.98.3:41703)
0927 02:47:41.923623 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.923625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.923633 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:47:41.923655 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.923655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.923666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.923714 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:47:41.923860 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.924608 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.926490 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.927105 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.927168 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.927177 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.927380 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.927419 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:47:41.927707 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.923485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41703 (local address 127.0.98.1:55933)
0927 02:47:41.923526 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:41.923538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:47:41.923579 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:47:41.923722 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:47:41.923723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:47:41.923723 (+ 0us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:47:41.923812 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.923813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.924688 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.924689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.925942 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:47:41.925945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.927192 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.927193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:47:41.927286 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.927290 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:47:41.927578 (+ 288us) client_negotiation.cc:240] Negotiation successful
0927 02:47:41.927623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":57}
I20260927 02:47:41.928193 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bfcf9d5c8f4979804706f1953e36b8" candidate_uuid: "2fbf107e71014a8e9e87c290dea96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" is_pre_election: true
I20260927 02:47:41.928359 7681 raft_consensus.cc:2454] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fbf107e71014a8e9e87c290dea96c46 in term 0.
I20260927 02:47:41.928578 7343 leader_election.cc:302] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbf107e71014a8e9e87c290dea96c46, 31f663fd4b464a1f9c1fcf19e24b8e8d; no voters:
I20260927 02:47:41.928567 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bfcf9d5c8f4979804706f1953e36b8" candidate_uuid: "2fbf107e71014a8e9e87c290dea96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4c314b85564149bb9e5664bc05515a" is_pre_election: true
I20260927 02:47:41.928789 7545 raft_consensus.cc:2454] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fbf107e71014a8e9e87c290dea96c46 in term 0.
I20260927 02:47:41.929337 7762 raft_consensus.cc:2781] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:47:41.929417 7762 raft_consensus.cc:482] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:47:41.929440 7762 raft_consensus.cc:3037] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:41.930327 7762 raft_consensus.cc:504] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.930563 7762 leader_election.cc:288] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [CANDIDATE]: Term 1 election: Requested vote from peers 31f663fd4b464a1f9c1fcf19e24b8e8d (127.0.98.3:41703), 7b4c314b85564149bb9e5664bc05515a (127.0.98.2:36541)
I20260927 02:47:41.930612 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bfcf9d5c8f4979804706f1953e36b8" candidate_uuid: "2fbf107e71014a8e9e87c290dea96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d"
I20260927 02:47:41.930708 7681 raft_consensus.cc:3037] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:41.930734 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bfcf9d5c8f4979804706f1953e36b8" candidate_uuid: "2fbf107e71014a8e9e87c290dea96c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4c314b85564149bb9e5664bc05515a"
I20260927 02:47:41.930827 7545 raft_consensus.cc:3037] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:47:41.931651 7545 raft_consensus.cc:2454] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbf107e71014a8e9e87c290dea96c46 in term 1.
I20260927 02:47:41.931869 7342 leader_election.cc:302] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fbf107e71014a8e9e87c290dea96c46, 7b4c314b85564149bb9e5664bc05515a; no voters:
I20260927 02:47:41.931993 7762 raft_consensus.cc:2781] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:47:41.932224 7762 raft_consensus.cc:686] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 1 LEADER]: Becoming Leader. State: Replica: 2fbf107e71014a8e9e87c290dea96c46, State: Running, Role: LEADER
I20260927 02:47:41.932340 7762 consensus_queue.cc:237] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } }
I20260927 02:47:41.932509 7681 raft_consensus.cc:2454] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fbf107e71014a8e9e87c290dea96c46 in term 1.
I20260927 02:47:41.933611 6492 catalog_manager.cc:5495] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fbf107e71014a8e9e87c290dea96c46 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2fbf107e71014a8e9e87c290dea96c46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fbf107e71014a8e9e87c290dea96c46" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 } health_report { overall_health: UNKNOWN } } }
I20260927 02:47:41.939006 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.936851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.98.1:43929)
0927 02:47:41.936890 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.936892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.936899 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:47:41.936965 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.936965 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.936980 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.937032 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:47:41.937120 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.937595 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.938373 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.938563 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.938612 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.938713 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:47:41.938801 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:47:41.938886 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.938916 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.938934 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:47:41.940685 7545 raft_consensus.cc:1261] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 1 FOLLOWER]: Refusing update from remote peer 2fbf107e71014a8e9e87c290dea96c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:41.940754 7681 raft_consensus.cc:1261] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 1 FOLLOWER]: Refusing update from remote peer 2fbf107e71014a8e9e87c290dea96c46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:47:41.941026 7763 consensus_queue.cc:1035] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4c314b85564149bb9e5664bc05515a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:41.941141 7762 consensus_queue.cc:1035] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31f663fd4b464a1f9c1fcf19e24b8e8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:47:41.943635 7766 mvcc.cc:205] Tried to move back new op lower bound from 7333794864906797056 to 7333794864875913216. Current Snapshot: MvccSnapshot[applied={T|T < 7333794864906797056}]
I20260927 02:47:41.944989 7767 mvcc.cc:205] Tried to move back new op lower bound from 7333794864906797056 to 7333794864875913216. Current Snapshot: MvccSnapshot[applied={T|T < 7333794864906797056}]
I20260927 02:47:41.945250 7769 mvcc.cc:205] Tried to move back new op lower bound from 7333794864906797056 to 7333794864875913216. Current Snapshot: MvccSnapshot[applied={T|T < 7333794864906797056}]
I20260927 02:47:41.947034 6491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:41.947118 6491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:47:41.948266 6491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00bb59c7500f4d4584e9398ff6f9097b: Sending DeleteTablet for 3 replicas of tablet f9bfcf9d5c8f4979804706f1953e36b8
I20260927 02:47:41.948611 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bfcf9d5c8f4979804706f1953e36b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47468
I20260927 02:47:41.948666 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bfcf9d5c8f4979804706f1953e36b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34258
I20260927 02:47:41.948889 7777 tablet_replica.cc:330] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: stopping tablet replica
I20260927 02:47:41.948985 7777 raft_consensus.cc:2229] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:41.949090 7777 raft_consensus.cc:2258] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:41.949079 7776 tablet_replica.cc:330] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: stopping tablet replica
I20260927 02:47:41.949184 7776 raft_consensus.cc:2229] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:47:41.949198 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bfcf9d5c8f4979804706f1953e36b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:47:41 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44878
I20260927 02:47:41.949317 7776 raft_consensus.cc:2258] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:41.949648 7777 ts_tablet_manager.cc:1905] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:41.949723 7776 ts_tablet_manager.cc:1905] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:47:41.950575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6460
I20260927 02:47:41.951360 7776 ts_tablet_manager.cc:1918] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:41.951421 7776 log.cc:1196] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/f9bfcf9d5c8f4979804706f1953e36b8
I20260927 02:47:41.951653 7776 ts_tablet_manager.cc:1939] T f9bfcf9d5c8f4979804706f1953e36b8 P 2fbf107e71014a8e9e87c290dea96c46: Deleting consensus metadata
I20260927 02:47:41.951695 7777 ts_tablet_manager.cc:1918] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:41.951752 7777 log.cc:1196] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/f9bfcf9d5c8f4979804706f1953e36b8
I20260927 02:47:41.951973 7777 ts_tablet_manager.cc:1939] T f9bfcf9d5c8f4979804706f1953e36b8 P 7b4c314b85564149bb9e5664bc05515a: Deleting consensus metadata
I20260927 02:47:41.954939 7778 tablet_replica.cc:330] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: stopping tablet replica
I20260927 02:47:41.955070 7778 raft_consensus.cc:2229] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:47:41.955209 7778 raft_consensus.cc:2258] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:47:41.955699 7778 ts_tablet_manager.cc:1905] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:47:41.955890 7481 connection.cc:535] server connection from 127.0.0.1:34258 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 02:47:41.956349 7343 connection.cc:535] server connection from 127.0.0.1:47468 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:47:41.957531 7778 ts_tablet_manager.cc:1918] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:47:41.957614 7778 log.cc:1196] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/f9bfcf9d5c8f4979804706f1953e36b8
I20260927 02:47:41.957862 7778 ts_tablet_manager.cc:1939] T f9bfcf9d5c8f4979804706f1953e36b8 P 31f663fd4b464a1f9c1fcf19e24b8e8d: Deleting consensus metadata
W20260927 02:47:41.958221 7614 connection.cc:441] server connection from 127.0.0.1:44878 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44878 (request call id 1) could send its response
I20260927 02:47:41.964461 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.957564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.0.98.61:46837)
0927 02:47:41.957611 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.957613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.957767 (+ 154us) server_negotiation.cc:407] Connection header received
0927 02:47:41.957869 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.957870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.957880 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.957908 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.958960 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.959664 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.960395 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.960626 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.960676 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.962209 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.962211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.962211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.962218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.963456 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.963791 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.963792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.963793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.963849 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.964010 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.964011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.964012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.964114 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.964202 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.964290 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.964369 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:47:41.965449 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:41.957500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56286 (local address 127.0.98.60:35181)
0927 02:47:41.957542 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:47:41.957544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:47:41.957814 (+ 270us) server_negotiation.cc:407] Connection header received
0927 02:47:41.957932 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:47:41.957934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:47:41.957945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:47:41.957973 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:47:41.958838 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.959616 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.960808 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:47:41.961051 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:47:41.961106 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:47:41.962231 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:47:41.962233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:47:41.962233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:47:41.962241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:47:41.964344 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.964734 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.964735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.964735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.964787 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:47:41.964981 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:47:41.964982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:47:41.964982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:47:41.965084 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:47:41.965154 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:47:41.965274 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 02:47:41.965364 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:47:42.945140 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.944871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:33179)
0927 02:47:42.945012 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.945084 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
W20260927 02:47:42.945411 7456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:42.945637 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.945238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:53085)
0927 02:47:42.945447 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.945595 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260927 02:47:42.945641 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:42.945148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:39589)
0927 02:47:42.945535 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:42.945599 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":51,"threads_started":1}
W20260927 02:47:42.945809 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
W20260927 02:47:42.945936 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:43.449183 7789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bb59c7500f4d4584e9398ff6f9097b)
I20260927 02:47:43.449357 7789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:43.449569 7789 leader_election.cc:288] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157)
I20260927 02:47:43.449922 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.449735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41822)
0927 02:47:43.449829 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.449884 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":45,"threads_started":1}
W20260927 02:47:43.450059 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:43.450436 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.450364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:41836)
0927 02:47:43.450399 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.450421 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:47:43.450505 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:43.450559 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 504a1aee5b9c4329aec80a8ecf97109a; no voters: 00bb59c7500f4d4584e9398ff6f9097b
I20260927 02:47:43.450652 7789 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:43.946139 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.945830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:36829)
0927 02:47:43.945983 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.946022 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260927 02:47:43.946398 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.946105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:43623)
0927 02:47:43.946329 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.946360 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":49,"threads_started":1}
I20260927 02:47:43.946416 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:43.946261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:53845)
0927 02:47:43.946345 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:43.946377 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":46,"threads_started":1}
I20260927 02:47:44.946794 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:44.946563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:56601)
0927 02:47:44.946685 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:44.946736 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":81,"threads_started":1}
I20260927 02:47:44.947067 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:44.946893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:34115)
0927 02:47:44.946997 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:44.947026 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260927 02:47:44.947090 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:44.946787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:48415)
0927 02:47:44.947005 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:44.947041 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260927 02:47:45.015295 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bb59c7500f4d4584e9398ff6f9097b)
I20260927 02:47:45.015406 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:45.015563 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157)
I20260927 02:47:45.015933 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.015761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:49990)
0927 02:47:45.015846 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.015893 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":44,"threads_started":1}
I20260927 02:47:45.016388 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.016269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:49996)
0927 02:47:45.016333 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.016363 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 02:47:45.016469 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:45.016526 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 504a1aee5b9c4329aec80a8ecf97109a; no voters: 00bb59c7500f4d4584e9398ff6f9097b
I20260927 02:47:45.016604 7798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:45.947661 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.947397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:49567)
0927 02:47:45.947566 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.947604 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":60,"threads_started":1}
I20260927 02:47:45.947785 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.947588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:56125)
0927 02:47:45.947706 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.947742 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260927 02:47:45.947789 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:45.947621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:58397)
0927 02:47:45.947733 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:45.947756 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260927 02:47:46.796833 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bb59c7500f4d4584e9398ff6f9097b)
I20260927 02:47:46.796928 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:46.797051 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157)
I20260927 02:47:46.797398 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.797199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:50188)
0927 02:47:46.797303 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.797340 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260927 02:47:46.797786 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.797703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:50196)
0927 02:47:46.797745 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.797769 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:47:46.797849 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:46.797900 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 504a1aee5b9c4329aec80a8ecf97109a; no voters: 00bb59c7500f4d4584e9398ff6f9097b
I20260927 02:47:46.797978 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:46.948379 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.948136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:53441)
0927 02:47:46.948291 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.948327 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260927 02:47:46.948379 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.948136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:43709)
0927 02:47:46.948283 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.948324 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260927 02:47:46.948515 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:46.948136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:44581)
0927 02:47:46.948433 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:46.948477 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":76,"threads_started":1}
I20260927 02:47:47.949005 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:47.948844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:39425)
0927 02:47:47.948948 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:47.948972 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":39,"threads_started":1}
I20260927 02:47:47.948982 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:47.948754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:33815)
0927 02:47:47.948891 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:47.948927 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260927 02:47:47.948983 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:47.948771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:58413)
0927 02:47:47.948884 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:47.948935 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260927 02:47:48.949641 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:48.949426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:50073)
0927 02:47:48.949549 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:48.949590 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260927 02:47:48.949661 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:48.949431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:59221)
0927 02:47:48.949570 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:48.949624 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260927 02:47:48.949983 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:48.949843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:48375)
0927 02:47:48.949927 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:48.949951 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":43,"threads_started":1}
I20260927 02:47:49.950312 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:49.950100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:58919)
0927 02:47:49.950224 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:49.950261 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260927 02:47:49.950321 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:49.950059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:49953)
0927 02:47:49.950224 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:49.950262 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":61,"threads_started":1}
I20260927 02:47:49.950443 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:49.950275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:58509)
0927 02:47:49.950379 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:49.950407 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260927 02:47:50.879258 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bb59c7500f4d4584e9398ff6f9097b)
I20260927 02:47:50.879350 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:50.879498 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157)
I20260927 02:47:50.879776 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.879616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:50530)
0927 02:47:50.879707 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.879742 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":44,"threads_started":1}
W20260927 02:47:50.879891 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260927 02:47:50.880303 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.880223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:50534)
0927 02:47:50.880263 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.880286 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:47:50.880367 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:50.880417 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 504a1aee5b9c4329aec80a8ecf97109a; no voters: 00bb59c7500f4d4584e9398ff6f9097b
I20260927 02:47:50.880486 7819 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:50.950966 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.950733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:56005)
0927 02:47:50.950864 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.950908 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
I20260927 02:47:50.951047 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.950730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:57229)
0927 02:47:50.950984 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.951013 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":86,"threads_started":1}
I20260927 02:47:50.951108 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:50.950730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:45399)
0927 02:47:50.951024 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:50.951063 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":56,"threads_started":1}
I20260927 02:47:51.951766 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:51.951517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:38367)
0927 02:47:51.951666 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:51.951710 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260927 02:47:51.951915 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:51.951517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:59431)
0927 02:47:51.951834 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:51.951869 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":65,"threads_started":1}
I20260927 02:47:51.952075 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:51.951741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:59603)
0927 02:47:51.952000 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:51.952033 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":47,"threads_started":1}
I20260927 02:47:52.518589 7828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bb59c7500f4d4584e9398ff6f9097b)
I20260927 02:47:52.518692 7828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:47:52.518903 7828 leader_election.cc:288] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157)
I20260927 02:47:52.519217 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.519045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:50656)
0927 02:47:52.519147 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.519182 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260927 02:47:52.519603 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.519513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:50662)
0927 02:47:52.519557 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.519583 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:47:52.519666 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:47:52.519706 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 504a1aee5b9c4329aec80a8ecf97109a; no voters: 00bb59c7500f4d4584e9398ff6f9097b
I20260927 02:47:52.519801 7828 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:47:52.952651 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.952405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:48677)
0927 02:47:52.952539 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.952578 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260927 02:47:52.952677 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.952519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:52209)
0927 02:47:52.952616 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.952642 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":43,"threads_started":1}
I20260927 02:47:52.952806 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:52.952597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:48279)
0927 02:47:52.952728 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:52.952762 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260927 02:47:53.953279 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:53.953057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:45693)
0927 02:47:53.953187 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:53.953223 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260927 02:47:53.953280 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:53.953079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:53481)
0927 02:47:53.953195 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:53.953233 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260927 02:47:53.953336 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:53.953179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:58525)
0927 02:47:53.953275 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:53.953299 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260927 02:47:54.953945 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:54.953664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:53719)
0927 02:47:54.953866 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:54.953893 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":60,"threads_started":1}
I20260927 02:47:54.953945 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:54.953763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:35603)
0927 02:47:54.953853 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:54.953901 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":43,"threads_started":1}
I20260927 02:47:54.954151 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:54.953840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:45699)
0927 02:47:54.954080 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:54.954114 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":66,"threads_started":1}
I20260927 02:47:55.954639 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:55.954378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:36307)
0927 02:47:55.954549 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:55.954586 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":61,"threads_started":1}
I20260927 02:47:55.954658 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:55.954461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:57053)
0927 02:47:55.954575 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:55.954610 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260927 02:47:55.954898 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:55.954520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:45703)
0927 02:47:55.954827 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:55.954860 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":44,"threads_started":1}
I20260927 02:47:56.955564 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:56.955305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:55755)
0927 02:47:56.955436 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:56.955481 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
I20260927 02:47:56.955567 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:56.955236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:59935)
0927 02:47:56.955483 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:56.955522 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":61,"threads_started":1}
I20260927 02:47:56.955648 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:56.955463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:46109)
0927 02:47:56.955584 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:56.955613 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":40,"threads_started":1}
I20260927 02:47:57.956194 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:57.955989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:40545)
0927 02:47:57.956103 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:57.956141 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260927 02:47:57.956326 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:57.956146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:54599)
0927 02:47:57.956250 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:57.956281 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260927 02:47:57.956329 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:57.955977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:52189)
0927 02:47:57.956263 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:57.956292 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":62,"threads_started":1}
I20260927 02:47:58.956844 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.956622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:33597)
0927 02:47:58.956746 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:58.956789 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260927 02:47:58.956993 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.956807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:42423)
0927 02:47:58.956913 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:58.956946 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260927 02:47:58.957199 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:58.957009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:59473)
0927 02:47:58.957128 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:58.957165 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260927 02:47:59.957759 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.957526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:46825)
0927 02:47:59.957658 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.957700 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260927 02:47:59.957768 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.957295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:42659)
0927 02:47:59.957682 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.957721 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":62,"threads_started":1}
I20260927 02:47:59.957829 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:47:59.957651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:34153)
0927 02:47:59.957762 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:47:59.957794 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
I20260927 02:48:00.881306 7855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 00bb59c7500f4d4584e9398ff6f9097b)
I20260927 02:48:00.881412 7855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "00bb59c7500f4d4584e9398ff6f9097b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42157 } } peers { permanent_uuid: "504a1aee5b9c4329aec80a8ecf97109a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46837 } attrs { promote: false } } peers { permanent_uuid: "654d3e37bffc41a08e73999db16af9e2" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35181 } attrs { promote: true } }
I20260927 02:48:00.881515 7855 leader_election.cc:288] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157)
I20260927 02:48:00.881794 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.881626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:35862)
0927 02:48:00.881727 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.881760 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
W20260927 02:48:00.881867 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260927 02:48:00.882177 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.882097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.0.1:35878)
0927 02:48:00.882137 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.882161 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:48:00.882243 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 00bb59c7500f4d4584e9398ff6f9097b (127.0.98.62:42157): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
I20260927 02:48:00.882287 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 504a1aee5b9c4329aec80a8ecf97109a; no voters: 00bb59c7500f4d4584e9398ff6f9097b
I20260927 02:48:00.882356 7855 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 504a1aee5b9c4329aec80a8ecf97109a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:48:00.958361 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.958156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:58409)
0927 02:48:00.958265 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.958305 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
I20260927 02:48:00.958478 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.958255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:43589)
0927 02:48:00.958400 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.958434 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":55,"threads_started":1}
I20260927 02:48:00.958478 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:00.958172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:34135)
0927 02:48:00.958415 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:00.958441 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":56,"threads_started":1}
I20260927 02:48:01.959183 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:01.958843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:57847)
0927 02:48:01.959087 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:01.959128 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":66,"threads_started":1}
I20260927 02:48:01.959195 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:01.958949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:40011)
0927 02:48:01.959111 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:01.959148 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":49,"threads_started":1}
I20260927 02:48:01.959223 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:01.959044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:50845)
0927 02:48:01.959163 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:01.959186 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":51,"threads_started":1}
I20260927 02:48:02.959893 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.959680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:60011)
0927 02:48:02.959803 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.959839 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260927 02:48:02.959896 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.959727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:60035)
0927 02:48:02.959813 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.959851 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":39,"threads_started":1}
I20260927 02:48:02.959910 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:02.959752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:54029)
0927 02:48:02.959851 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:02.959875 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260927 02:48:03.960566 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.960343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:37589)
0927 02:48:03.960472 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.960510 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260927 02:48:03.960637 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.960464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:41367)
0927 02:48:03.960558 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.960591 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":41,"threads_started":1}
I20260927 02:48:03.960664 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:03.960342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:44525)
0927 02:48:03.960595 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:03.960627 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":70,"threads_started":1}
I20260927 02:48:04.961258 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:04.961024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:46105)
0927 02:48:04.961164 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:04.961206 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260927 02:48:04.961259 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:04.961045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:49145)
0927 02:48:04.961164 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:04.961206 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":50,"threads_started":1}
I20260927 02:48:04.961258 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:04.961024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:35899)
0927 02:48:04.961166 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:04.961206 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260927 02:48:05.961951 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.961718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:60499)
0927 02:48:05.961855 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:05.961897 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260927 02:48:05.961951 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.961743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:55329)
0927 02:48:05.961858 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:05.961897 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260927 02:48:05.962101 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:05.961716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:55141)
0927 02:48:05.961978 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:05.962009 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":80,"threads_started":1}
I20260927 02:48:06.962639 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.962417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:36285)
0927 02:48:06.962542 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.962583 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260927 02:48:06.962769 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.962429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:43605)
0927 02:48:06.962666 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.962697 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":49,"threads_started":1}
I20260927 02:48:06.962775 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:06.962571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:51251)
0927 02:48:06.962692 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:06.962726 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":53,"threads_started":1}
I20260927 02:48:07.963259 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.963057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:48019)
0927 02:48:07.963180 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.963215 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260927 02:48:07.963263 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.963057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:56617)
0927 02:48:07.963182 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.963216 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260927 02:48:07.963405 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:07.963232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:54913)
0927 02:48:07.963336 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:07.963369 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260927 02:48:08.963820 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:08.963601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:38895)
0927 02:48:08.963723 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:08.963766 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":87,"threads_started":1}
I20260927 02:48:08.963827 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:08.963646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:39385)
0927 02:48:08.963763 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:08.963793 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":50,"threads_started":1}
I20260927 02:48:08.963848 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:08.963701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:60641)
0927 02:48:08.963779 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:08.963812 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63,"thread_start_us":41,"threads_started":1}
I20260927 02:48:09.964378 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:09.964172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:58319)
0927 02:48:09.964305 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:09.964331 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260927 02:48:09.964378 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:09.964203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:33465)
0927 02:48:09.964294 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:09.964331 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":41,"threads_started":1}
I20260927 02:48:09.964392 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:09.964164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:43409)
0927 02:48:09.964292 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:09.964331 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 02:48:10.964964 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:10.964760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:60411)
0927 02:48:10.964893 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:10.964925 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260927 02:48:10.964957 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:10.964755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:34583)
0927 02:48:10.964875 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:10.964913 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260927 02:48:10.965142 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:10.964753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:56239)
0927 02:48:10.965072 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:10.965104 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":71,"threads_started":1}
I20260927 02:48:11.965596 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:11.965369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.1:60613)
0927 02:48:11.965483 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:11.965517 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260927 02:48:11.965590 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:11.965452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.3:52727)
0927 02:48:11.965534 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:11.965556 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":38,"threads_started":1}
I20260927 02:48:11.965587 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:11.965401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42157 (local address 127.0.98.2:51629)
0927 02:48:11.965506 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:11.965540 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:42157,127.0.98.61:46837,127.0.98.60:35181, num_attempts: 1)
I20260927 02:48:12.107970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 7326
I20260927 02:48:12.113015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 7462
I20260927 02:48:12.116910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 7598
I20260927 02:48:12.120920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 6956
I20260927 02:48:12.124650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 7143
2026-09-27T02:48:12Z chronyd exiting
I20260927 02:48:12.139300 392 test_util.cc:175] -----------------------------------------------
I20260927 02:48:12.139355 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790477232139228-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (32869 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:12 dist-test-slave-f4rg krb5kdc[7896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:12 dist-test-slave-f4rg krb5kdc[7896](info): set up 2 sockets
Sep 27 02:48:12 dist-test-slave-f4rg krb5kdc[7896](info): commencing operation
krb5kdc: starting...
W20260927 02:48:13.082944 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:13Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 02:48:13.191797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34067
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:13.269933 7909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:13.270138 7909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:13.270166 7909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:13.271332 7909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:13.271376 7909 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 02:48:13.271392 7909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:13.271410 7909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:13.271425 7909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:13.271440 7909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:13.272869 7909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34067
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:13.273103 7909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:13.273304 7909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:13.278064 7917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:13.278208 7914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:13.278273 7909 server_base.cc:1034] running on GCE node
W20260927 02:48:13.278070 7915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:13.278566 7909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:13.278872 7909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:13.280022 7909 hybrid_clock.cc:648] HybridClock initialized: now 1790477293280015 us; error 33 us; skew 500 ppm
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:13.282401 7909 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 02:48:13.282852 7909 webserver.cc:460] Webserver started at http://127.0.98.62:41629/ using document root <none> and password file <none>
I20260927 02:48:13.283098 7909 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:13.283142 7909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:13.283270 7909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:13.284287 7909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "1d4b79e59fdfc82965b8f85bc295492f"
server_key_iv: "31564c55c9791e309d34aa40251cd878"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.284601 7909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "1d4b79e59fdfc82965b8f85bc295492f"
server_key_iv: "31564c55c9791e309d34aa40251cd878"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.285804 7909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:48:13.286520 7924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.286687 7909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:13.286768 7909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "1d4b79e59fdfc82965b8f85bc295492f"
server_key_iv: "31564c55c9791e309d34aa40251cd878"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.286861 7909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:13.297187 7909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:13.298942 7909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:13.299098 7909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:13.303395 7909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34067
I20260927 02:48:13.303442 7986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34067 every 8 connection(s)
I20260927 02:48:13.303732 7909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:13.304320 7987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.305708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 7909
I20260927 02:48:13.305800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:13.305974 392 external_mini_cluster.cc:1428] Setting key 376153cfb5f5e2034f92d271e8bf6305
I20260927 02:48:13.306664 7987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Bootstrap starting.
I20260927 02:48:13.307233 7987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.307485 7987 log.cc:824] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:13.308061 7987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: No bootstrap required, opened a new log
I20260927 02:48:13.309774 7987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260927 02:48:13.309906 7987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.309935 7987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d4dc0acc8d42f1a3b0fd1875f87ee3, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.310058 7987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260927 02:48:13.310134 7987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:13.310189 7987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:13.310241 7987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:13.310892 7987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260927 02:48:13.310992 7987 leader_election.cc:302] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67d4dc0acc8d42f1a3b0fd1875f87ee3; no voters:
I20260927 02:48:13.311143 7987 leader_election.cc:288] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:13.311198 7992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:13.311378 7987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:13.311403 7992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [term 1 LEADER]: Becoming Leader. State: Replica: 67d4dc0acc8d42f1a3b0fd1875f87ee3, State: Running, Role: LEADER
I20260927 02:48:13.311501 7992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } }
I20260927 02:48:13.311822 7994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67d4dc0acc8d42f1a3b0fd1875f87ee3. Latest consensus state: current_term: 1 leader_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } } }
I20260927 02:48:13.312039 7994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:13.311980 7993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67d4dc0acc8d42f1a3b0fd1875f87ee3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34067 } } }
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:48:13.312276 7993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:13.312381 7996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:13.312850 7996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:13.314322 7996 catalog_manager.cc:1347] Generated new cluster ID: de282673714244a39deac9d7541b622b
I20260927 02:48:13.314373 7996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:13.316082 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.306476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.0.98.62:34067)
0927 02:48:13.306676 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.306684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.306710 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:48:13.307181 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.307189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.307235 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.307359 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.308037 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.308950 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.309650 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.310092 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.310186 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.312796 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.312799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.312800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.312812 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.314715 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.315063 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.315066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.315067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.315158 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.315323 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.315324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.315324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.315547 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.315681 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.315790 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.315915 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260927 02:48:13.322540 7996 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:13.323082 7996 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:13.328481 7996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Generated new TSK 0
I20260927 02:48:13.328634 7996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 02:48:13.355716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:13.440536 8013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:13.440735 8013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:13.440757 8013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:13.441905 8013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:13.441947 8013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:13.441967 8013 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:48:13.441983 8013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:13.443444 8013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:13.443652 8013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:13.443858 8013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:13.448536 8019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:13.448541 8021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:13.448540 8018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:13.448673 8013 server_base.cc:1034] running on GCE node
I20260927 02:48:13.448982 8013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:13.449219 8013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:13.450349 8013 hybrid_clock.cc:648] HybridClock initialized: now 1790477293450338 us; error 28 us; skew 500 ppm
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:13.452831 8013 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:48:13.453285 8013 webserver.cc:460] Webserver started at http://127.0.98.1:33657/ using document root <none> and password file <none>
I20260927 02:48:13.453495 8013 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:13.453542 8013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:13.453668 8013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:13.454703 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2e20d67cfe64284b5039cde3c098666"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "9aa1b11890c5e51607973f1dd7f265cb"
server_key_iv: "97997be69aebb7ae9879bc509a95719d"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.455073 8013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2e20d67cfe64284b5039cde3c098666"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "9aa1b11890c5e51607973f1dd7f265cb"
server_key_iv: "97997be69aebb7ae9879bc509a95719d"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.456281 8013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:13.456887 8028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.457036 8013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:13.457115 8013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "e2e20d67cfe64284b5039cde3c098666"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "9aa1b11890c5e51607973f1dd7f265cb"
server_key_iv: "97997be69aebb7ae9879bc509a95719d"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.457180 8013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:13.473244 8013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:13.474999 8013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:13.475142 8013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:13.475600 8013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:13.475638 8013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.475665 8013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:13.475682 8013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.481724 8013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41539
I20260927 02:48:13.481841 8143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41539 every 8 connection(s)
I20260927 02:48:13.482120 8013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:48:13.484849 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.476035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51917 (local address 127.0.98.62:34067)
0927 02:48:13.476076 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.476078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.476291 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:48:13.476792 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.476793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.476806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.476856 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.477630 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.478379 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.479881 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.480147 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.480202 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.482704 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.482705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.482705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.482711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.483833 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.484158 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.484159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.484159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.484204 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.484365 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.484365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.484365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.484454 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.484497 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.484728 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.484788 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:48:13.484998 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.476038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.1:51917)
0927 02:48:13.476181 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.476214 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.476692 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.476937 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.476941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.477278 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:13.477544 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.477550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.478484 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.478486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.479794 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.479798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.480234 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.480235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.480385 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.480889 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:13.480901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:13.482563 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:13.483895 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.483900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.483902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.484098 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.484243 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.484244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.484244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.484326 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.484496 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:13.484498 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:13.484634 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.484713 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.484824 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260927 02:48:13.485419 8144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34067
I20260927 02:48:13.485494 8144 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:13.485661 8144 heartbeater.cc:511] Master 127.0.98.62:34067 requested a full tablet report, sending...
I20260927 02:48:13.486166 7940 ts_manager.cc:194] Registered new tserver with Master: e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539)
I20260927 02:48:13.487020 7940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51917
I20260927 02:48:13.490500 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 8013
I20260927 02:48:13.490613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:13.490846 392 external_mini_cluster.cc:1428] Setting key b08b9b32baefcf3c2dbd1537fdd84fe1
I20260927 02:48:13.492647 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.486347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46837 (local address 127.0.98.62:34067)
0927 02:48:13.486384 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.486386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.486583 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:48:13.486652 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.486653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.486662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.486717 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.487661 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.488267 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.489206 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.489470 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.489528 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.490301 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.490302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.490303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.490309 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.491744 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.492025 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.492026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.492027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.492081 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.492253 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.492254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.492254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.492365 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.492422 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.492507 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.492568 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:48:13.492658 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.486341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.1:46837)
0927 02:48:13.486508 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.486519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.486581 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.487320 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.487321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.487509 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:13.487571 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.487573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.488358 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.488359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.489107 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.489110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.489526 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.489527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.489583 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.489958 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:13.489963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:13.490227 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:13.491817 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.491817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.491818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.491968 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.492161 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.492161 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.492162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.492205 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.492413 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:13.492414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:13.492442 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.492485 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.492553 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 02:48:13.513603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:13.592172 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:13.592371 8153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:13.592402 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:13.593539 8153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:13.593580 8153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:13.593607 8153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:48:13.593629 8153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:13.595091 8153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:13.595326 8153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:13.595532 8153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:13.600335 8158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:13.600312 8159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:13.600337 8153 server_base.cc:1034] running on GCE node
W20260927 02:48:13.600312 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:13.600710 8153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:13.600900 8153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:13.602043 8153 hybrid_clock.cc:648] HybridClock initialized: now 1790477293602029 us; error 33 us; skew 500 ppm
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:13.604429 8153 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:48:13.604822 8153 webserver.cc:460] Webserver started at http://127.0.98.2:36037/ using document root <none> and password file <none>
I20260927 02:48:13.605024 8153 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:13.605070 8153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:13.605197 8153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:13.606204 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "675c67f791a0946eb53c6692008e5612"
server_key_iv: "8a9fb898aa48a36684edc630abb648f7"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.606527 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "675c67f791a0946eb53c6692008e5612"
server_key_iv: "8a9fb898aa48a36684edc630abb648f7"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.607822 8153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:48:13.608458 8168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.608629 8153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:13.608697 8153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "675c67f791a0946eb53c6692008e5612"
server_key_iv: "8a9fb898aa48a36684edc630abb648f7"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.608757 8153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:13.621186 8153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:13.623275 8153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:13.623404 8153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:13.623857 8153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:13.623899 8153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.623929 8153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:13.623953 8153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.630266 8153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39471
I20260927 02:48:13.630409 8283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39471 every 8 connection(s)
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:48:13.630640 8153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:13.633257 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.624303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59737 (local address 127.0.98.62:34067)
0927 02:48:13.624367 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.624369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.624561 (+ 192us) server_negotiation.cc:407] Connection header received
0927 02:48:13.625028 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.625030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.625043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.625121 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.625851 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.626613 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.628129 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.628435 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.628506 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.630986 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.630987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.630987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.630995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.632232 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.632558 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.632559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.632559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.632635 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.632797 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.632798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.632798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.632891 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.632933 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.633140 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.633195 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:48:13.633428 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.624277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.2:59737)
0927 02:48:13.624437 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.624476 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.624944 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.625177 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.625181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.625491 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:13.625771 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.625777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.626711 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.626713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.628044 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.628048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.628512 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.628513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.628664 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.629219 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:13.629230 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:13.630881 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:13.632299 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.632303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.632304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.632501 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.632677 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.632677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.632678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.632758 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.632933 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:13.632936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:13.633045 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.633128 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.633241 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260927 02:48:13.633880 8284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34067
I20260927 02:48:13.633955 8284 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:13.634126 8284 heartbeater.cc:511] Master 127.0.98.62:34067 requested a full tablet report, sending...
I20260927 02:48:13.634481 7941 ts_manager.cc:194] Registered new tserver with Master: c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471)
I20260927 02:48:13.635028 7941 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59737
I20260927 02:48:13.637737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 8153
I20260927 02:48:13.637835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:13.638021 392 external_mini_cluster.cc:1428] Setting key 4d764dddbb8abe449f164cb82aa47c38
I20260927 02:48:13.640311 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.635384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55657 (local address 127.0.98.62:34067)
0927 02:48:13.635430 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.635431 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.635437 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:48:13.635452 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.635453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.635461 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.635505 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.635801 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.636265 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.636913 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.637107 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.637163 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.637904 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.637905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.637905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.637911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.639318 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.639595 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.639596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.639596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.639650 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.639809 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.639809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.639810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.639977 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.640044 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.640189 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.640253 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:48:13.640353 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.634887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.2:55657)
0927 02:48:13.634997 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.635006 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.635048 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.635518 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.635519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.635684 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:13.635736 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.635738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.636348 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.636349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.636848 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.636849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.637168 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.637169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.637228 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.637558 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:13.637561 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:13.637826 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:13.639386 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.639387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.639387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.639535 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.639703 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.639704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.639704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.639761 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.640047 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:13.640047 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:13.640093 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.640163 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.640256 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:48:13.661247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34067
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:13.739804 8293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:13.740008 8293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:13.740041 8293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:13.741194 8293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:13.741236 8293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:13.741263 8293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 02:48:13.741286 8293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:13.742718 8293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:13.742969 8293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:13.743177 8293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:13.747829 8298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:13.747821 8299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:13.747821 8301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:13.748013 8293 server_base.cc:1034] running on GCE node
I20260927 02:48:13.748198 8293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:13.748425 8293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:13.749553 8293 hybrid_clock.cc:648] HybridClock initialized: now 1790477293749542 us; error 27 us; skew 500 ppm
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:13.751888 8293 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:48:13.752298 8293 webserver.cc:460] Webserver started at http://127.0.98.3:40279/ using document root <none> and password file <none>
I20260927 02:48:13.752503 8293 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:13.752552 8293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:13.752679 8293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:13.753715 8293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "ebd7b29deddaf42323032aa289e94d1f"
server_key_iv: "b29d9078ff8d03990990a68495118251"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.754031 8293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "ebd7b29deddaf42323032aa289e94d1f"
server_key_iv: "b29d9078ff8d03990990a68495118251"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.755327 8293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:48:13.756018 8308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.756160 8293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:13.756235 8293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
format_stamp: "Formatted at 2026-09-27 02:48:13 on dist-test-slave-f4rg"
server_key: "ebd7b29deddaf42323032aa289e94d1f"
server_key_iv: "b29d9078ff8d03990990a68495118251"
server_key_version: "encryptionkey@0"
I20260927 02:48:13.756291 8293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:13.780732 8293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:13.782945 8293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:13.783085 8293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:13.783644 8293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:13.783695 8293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.783725 8293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:13.783747 8293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:48:13.790067 8293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42109
I20260927 02:48:13.790165 8423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42109 every 8 connection(s)
I20260927 02:48:13.790457 8293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:13.792341 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.784021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45843 (local address 127.0.98.62:34067)
0927 02:48:13.784072 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.784074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.784293 (+ 219us) server_negotiation.cc:407] Connection header received
0927 02:48:13.784773 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.784775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.784786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.784878 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.785658 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.786404 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.787547 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.787755 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.787810 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.789906 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.789907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.789907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.789915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.791274 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.791624 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.791625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.791625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.791672 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.791843 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.791844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.791844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.791948 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.791997 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.792221 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.792278 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:48:13.792507 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.784012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.3:45843)
0927 02:48:13.784170 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.784210 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.784688 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.784960 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.784963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.785275 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:13.785580 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.785586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.786504 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.786505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.787467 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.787470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.787837 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.787838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.787957 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.788425 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:13.788433 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:13.789819 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:13.791340 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.791344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.791346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.791551 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.791707 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.791708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.791708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.791793 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.791995 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:13.791998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:13.792128 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.792204 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.792319 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":97,"threads_started":1}
I20260927 02:48:13.792930 8424 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34067
I20260927 02:48:13.793006 8424 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:13.793171 8424 heartbeater.cc:511] Master 127.0.98.62:34067 requested a full tablet report, sending...
I20260927 02:48:13.793542 7940 ts_manager.cc:194] Registered new tserver with Master: 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:13.794021 7940 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45843
I20260927 02:48:13.795775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 8293
I20260927 02:48:13.795883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:13.796074 392 external_mini_cluster.cc:1428] Setting key c1fd98b7c7f0de0909290088a3c36735
I20260927 02:48:13.796931 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:48:13.799244 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.793892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46753 (local address 127.0.98.62:34067)
0927 02:48:13.793927 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.793929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.793971 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:48:13.794032 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.794032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.794040 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.794084 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.794383 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.795014 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.795702 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.795971 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.796046 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.797044 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.797046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.797046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.797053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.798295 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.798564 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.798564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.798565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.798626 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.798831 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.798831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.798832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.798939 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.798987 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.799132 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.799186 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:48:13.799307 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.793817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.98.3:46753)
0927 02:48:13.793920 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.793930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.793984 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.794088 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.794089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.794267 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:13.794323 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.794325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.795098 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.795099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.795622 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.795624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.796041 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.796041 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.796112 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.796594 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:13.796599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:13.796962 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:13.798358 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.798359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.798359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.798506 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.798682 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:13.798682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:13.798683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:13.798746 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:13.798991 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:13.798991 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:13.799034 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.799112 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.799204 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":57,"threads_started":1}
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:48:13.817070 7990 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:13.811801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.0.98.62:34067)
0927 02:48:13.811854 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.811856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.811865 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:13.811914 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.811915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.811926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.811977 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.812626 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.813166 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.813990 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.814186 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.814244 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.816988 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45132: BlockingRecv error: recv got EOF from 127.0.0.1:45132 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 02:48:13 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:48:13.823650 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.817762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45134 (local address 127.0.98.62:34067)
0927 02:48:13.817814 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.817816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.817865 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:48:13.817904 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.817905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.817914 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.817957 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:13.818341 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.818795 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.819444 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.819620 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.819663 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.821479 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:13.821480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:13.821480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:13.821486 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:13.822607 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.822928 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.822928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.822929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.822982 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:13.823166 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:13.823167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:13.823168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:13.823287 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:13.823370 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.823489 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.823558 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:48:13.824992 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:13.826462 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:13.832559 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.828519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41539 (local address 127.0.0.1:35452)
0927 02:48:13.828714 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.828723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.828761 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.829304 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.829305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.829307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:13.829354 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.829355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.830527 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.830528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.831516 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.831518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.832333 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.832334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.832386 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.832397 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.832446 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.832483 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260927 02:48:13.832612 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.828677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.98.1:41539)
0927 02:48:13.829101 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.829105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.829114 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:13.829148 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.829150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.829164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.829233 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:13.829438 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.830472 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.831585 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.832261 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.832336 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.832408 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.832473 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.832510 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":71,"threads_started":1}
I20260927 02:48:13.832872 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.828347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39471 (local address 127.0.0.1:37584)
0927 02:48:13.828466 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.828505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.828569 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.829000 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.829003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.829004 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:13.829067 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.829069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.830446 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.830446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.831701 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.831703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.832513 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.832514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.832565 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.832568 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.832770 (+ 202us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.832808 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":44,"thread_start_us":68,"threads_started":1}
I20260927 02:48:13.833112 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.828456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.0.98.2:39471)
0927 02:48:13.828847 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.828852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.828862 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:48:13.828898 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.828900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.828916 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.828990 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:13.829171 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.830370 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.831779 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.832444 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.832505 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.832571 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.832981 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.833023 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":80,"threads_started":1}
I20260927 02:48:13.833242 8078 tablet_service.cc:1467] Processing CreateTablet for tablet 846aaa4944b749e180a93adf70e4dfeb (DEFAULT_TABLE table=test-table [id=5442674d17ca41c99bbf0643252dfe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:13.833648 8078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846aaa4944b749e180a93adf70e4dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.833747 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.828647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.0.1:39988)
0927 02:48:13.828794 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.828804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.828853 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.829539 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.829540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.829541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:13.829571 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.829572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.830362 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.830363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.831702 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.831703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.833481 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.833482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.833542 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.833546 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.833601 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.833642 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":36,"thread_start_us":37,"threads_started":1}
I20260927 02:48:13.833779 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.828792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.0.98.3:42109)
0927 02:48:13.829350 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.829354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.829363 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:13.829397 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.829398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.829413 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.829479 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:13.829630 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.830291 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.832611 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.833296 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.833363 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.833445 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.833626 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.833685 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":77,"threads_started":1}
I20260927 02:48:13.834448 8354 tablet_service.cc:1467] Processing CreateTablet for tablet 846aaa4944b749e180a93adf70e4dfeb (DEFAULT_TABLE table=test-table [id=5442674d17ca41c99bbf0643252dfe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:13.834733 8354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846aaa4944b749e180a93adf70e4dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.835220 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 846aaa4944b749e180a93adf70e4dfeb (DEFAULT_TABLE table=test-table [id=5442674d17ca41c99bbf0643252dfe9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:13.835533 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846aaa4944b749e180a93adf70e4dfeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.836781 8449 tablet_bootstrap.cc:492] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Bootstrap starting.
I20260927 02:48:13.837348 8449 tablet_bootstrap.cc:654] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.837625 8449 log.cc:824] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:13.838136 8451 tablet_bootstrap.cc:492] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Bootstrap starting.
I20260927 02:48:13.838430 8449 tablet_bootstrap.cc:492] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: No bootstrap required, opened a new log
I20260927 02:48:13.838505 8449 ts_tablet_manager.cc:1397] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:13.838793 8451 tablet_bootstrap.cc:654] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.838958 8452 tablet_bootstrap.cc:492] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Bootstrap starting.
I20260927 02:48:13.839311 8451 log.cc:824] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:13.839555 8452 tablet_bootstrap.cc:654] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.839851 8452 log.cc:824] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:13.840086 8451 tablet_bootstrap.cc:492] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: No bootstrap required, opened a new log
I20260927 02:48:13.840153 8451 ts_tablet_manager.cc:1397] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:48:13.840435 8452 tablet_bootstrap.cc:492] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: No bootstrap required, opened a new log
I20260927 02:48:13.840489 8452 ts_tablet_manager.cc:1397] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:13.840471 8449 raft_consensus.cc:348] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.840652 8449 raft_consensus.cc:374] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.840691 8449 raft_consensus.cc:729] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.840798 8449 consensus_queue.cc:260] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.841053 8449 ts_tablet_manager.cc:1428] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 02:48:13.841084 8144 heartbeater.cc:503] Master 127.0.98.62:34067 was elected leader, sending a full tablet report...
I20260927 02:48:13.841988 8451 raft_consensus.cc:348] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.842144 8451 raft_consensus.cc:374] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.842186 8451 raft_consensus.cc:729] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863f3b4e4322499d81e8d4a6d1afa8e7, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.842218 8452 raft_consensus.cc:348] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.842299 8451 consensus_queue.cc:260] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.842370 8452 raft_consensus.cc:374] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.842399 8452 raft_consensus.cc:729] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c352abe2f4c942f2acd4cd38bd8c3b2e, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.842512 8452 consensus_queue.cc:260] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.842545 8451 ts_tablet_manager.cc:1428] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 02:48:13.842564 8424 heartbeater.cc:503] Master 127.0.98.62:34067 was elected leader, sending a full tablet report...
I20260927 02:48:13.842759 8452 ts_tablet_manager.cc:1428] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:13.842787 8284 heartbeater.cc:503] Master 127.0.98.62:34067 was elected leader, sending a full tablet report...
I20260927 02:48:13.844738 8457 raft_consensus.cc:482] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:13.844813 8457 raft_consensus.cc:504] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.845022 8457 leader_election.cc:288] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:13.847868 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.845326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60461 (local address 127.0.98.3:42109)
0927 02:48:13.845467 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.845468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.845476 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:48:13.845610 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.845611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.845620 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.845645 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:13.845794 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.846247 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.847242 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.847584 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.847648 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.847657 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.847778 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.847808 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 02:48:13.847887 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.845452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38063 (local address 127.0.98.1:41539)
0927 02:48:13.845485 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.845487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.845494 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:48:13.845513 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.845514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.845524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.845570 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:13.845784 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.846247 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.847246 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.847597 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.847655 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.847664 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.847791 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.847826 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:48:13.847918 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.845233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.98.2:60461)
0927 02:48:13.845414 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.845423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.845533 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.845696 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.845696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.845698 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:13.845749 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.845751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.846313 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.846314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.847171 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.847173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.847652 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.847653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.847722 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.847726 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.847769 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.847804 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":29,"thread_start_us":56,"threads_started":1}
I20260927 02:48:13.847934 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.845155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41539 (local address 127.0.98.2:38063)
0927 02:48:13.845190 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:13.845201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:13.845241 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:13.845622 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:13.845623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:13.845624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:13.845684 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.845687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.846340 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.846341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.847172 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:13.847174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.847660 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.847660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:13.847737 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.847740 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:13.847782 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:48:13.847821 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:48:13.848213 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846aaa4944b749e180a93adf70e4dfeb" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e20d67cfe64284b5039cde3c098666" is_pre_election: true
I20260927 02:48:13.848215 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846aaa4944b749e180a93adf70e4dfeb" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" is_pre_election: true
I20260927 02:48:13.848315 8378 raft_consensus.cc:2454] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 0.
I20260927 02:48:13.848325 8098 raft_consensus.cc:2454] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 0.
I20260927 02:48:13.848502 8172 leader_election.cc:302] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c352abe2f4c942f2acd4cd38bd8c3b2e, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:13.848613 8457 raft_consensus.cc:2781] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:13.848672 8457 raft_consensus.cc:482] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:13.848701 8457 raft_consensus.cc:3037] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:13.849555 8457 raft_consensus.cc:504] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.849668 8457 leader_election.cc:288] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 election: Requested vote from peers e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:13.849803 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846aaa4944b749e180a93adf70e4dfeb" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e20d67cfe64284b5039cde3c098666"
I20260927 02:48:13.849803 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846aaa4944b749e180a93adf70e4dfeb" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
I20260927 02:48:13.849869 8378 raft_consensus.cc:3037] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:13.849870 8098 raft_consensus.cc:3037] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:13.850476 8378 raft_consensus.cc:2454] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 1.
I20260927 02:48:13.850482 8098 raft_consensus.cc:2454] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 1.
I20260927 02:48:13.850632 8172 leader_election.cc:302] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c352abe2f4c942f2acd4cd38bd8c3b2e, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:13.850716 8457 raft_consensus.cc:2781] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:13.850881 8457 raft_consensus.cc:686] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 LEADER]: Becoming Leader. State: Replica: c352abe2f4c942f2acd4cd38bd8c3b2e, State: Running, Role: LEADER
I20260927 02:48:13.850962 8457 consensus_queue.cc:237] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.851666 7940 catalog_manager.cc:5495] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260927 02:48:13.858227 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:13.856411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.0.98.2:39471)
0927 02:48:13.856455 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:13.856457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:13.856463 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:48:13.856486 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:13.856486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:13.856500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:13.856551 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:13.856631 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.857054 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.857695 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:13.857863 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:13.857906 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:13.858005 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:13.858088 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:13.858150 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:13.858163 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 02:48:13.858178 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:48:13.859865 8378 raft_consensus.cc:1261] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Refusing update from remote peer c352abe2f4c942f2acd4cd38bd8c3b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:13.859882 8098 raft_consensus.cc:1261] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Refusing update from remote peer c352abe2f4c942f2acd4cd38bd8c3b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:13.860194 8459 consensus_queue.cc:1035] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:13.860288 8457 consensus_queue.cc:1035] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:13.864188 8464 mvcc.cc:205] Tried to move back new op lower bound from 7333794995647606784 to 7333794995614130176. Current Snapshot: MvccSnapshot[applied={T|T < 7333794995647606784 or (T in {7333794995647606784})}]
I20260927 02:48:13.864262 7940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:13.864333 7940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:13.864364 8463 mvcc.cc:205] Tried to move back new op lower bound from 7333794995647606784 to 7333794995614130176. Current Snapshot: MvccSnapshot[applied={T|T < 7333794995647606784 or (T in {7333794995647606784})}]
I20260927 02:48:13.864966 7940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Sending DeleteTablet for 3 replicas of tablet 846aaa4944b749e180a93adf70e4dfeb
I20260927 02:48:13.865238 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet 846aaa4944b749e180a93adf70e4dfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37584
I20260927 02:48:13.865314 8078 tablet_service.cc:1514] Processing DeleteTablet for tablet 846aaa4944b749e180a93adf70e4dfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35452
I20260927 02:48:13.865358 8354 tablet_service.cc:1514] Processing DeleteTablet for tablet 846aaa4944b749e180a93adf70e4dfeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:13 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39988
I20260927 02:48:13.865525 8471 tablet_replica.cc:330] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: stopping tablet replica
I20260927 02:48:13.865631 8471 raft_consensus.cc:2229] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:13.865628 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:48:13.865725 8471 raft_consensus.cc:2258] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:48:13.865779 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:13.866166 8472 tablet_replica.cc:330] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: stopping tablet replica
I20260927 02:48:13.866329 8471 ts_tablet_manager.cc:1905] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:13.866341 8472 raft_consensus.cc:2229] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:13.866520 8472 raft_consensus.cc:2258] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:13.867004 8472 ts_tablet_manager.cc:1905] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:13.867406 8078 tablet_service.cc:1467] Processing CreateTablet for tablet 9493fe09f69740cc945f34f9a942f072 (DEFAULT_TABLE table=test-table [id=00acb60a9bed49a888ffd6799e666b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:13.867512 8078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9493fe09f69740cc945f34f9a942f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.867909 8473 tablet_replica.cc:330] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: stopping tablet replica
I20260927 02:48:13.868011 8473 raft_consensus.cc:2229] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:13.868131 8471 ts_tablet_manager.cc:1918] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:13.868146 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 9493fe09f69740cc945f34f9a942f072 (DEFAULT_TABLE table=test-table [id=00acb60a9bed49a888ffd6799e666b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:13.868196 8471 log.cc:1196] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/846aaa4944b749e180a93adf70e4dfeb
I20260927 02:48:13.868235 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9493fe09f69740cc945f34f9a942f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.868449 8471 ts_tablet_manager.cc:1939] T 846aaa4944b749e180a93adf70e4dfeb P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting consensus metadata
I20260927 02:48:13.868813 8449 tablet_bootstrap.cc:492] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Bootstrap starting.
I20260927 02:48:13.868875 7928 catalog_manager.cc:4841] TS c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471): tablet 846aaa4944b749e180a93adf70e4dfeb (table test-table [id=5442674d17ca41c99bbf0643252dfe9e]) successfully deleted
I20260927 02:48:13.869177 8449 tablet_bootstrap.cc:654] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.869263 8473 raft_consensus.cc:2258] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:13.869354 8452 tablet_bootstrap.cc:492] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Bootstrap starting.
I20260927 02:48:13.869426 8354 tablet_service.cc:1467] Processing CreateTablet for tablet 9493fe09f69740cc945f34f9a942f072 (DEFAULT_TABLE table=test-table [id=00acb60a9bed49a888ffd6799e666b46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:13.869518 8354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9493fe09f69740cc945f34f9a942f072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:13.869728 8452 tablet_bootstrap.cc:654] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.869874 8473 ts_tablet_manager.cc:1905] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:13.870397 8472 ts_tablet_manager.cc:1918] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:13.870508 8472 log.cc:1196] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/846aaa4944b749e180a93adf70e4dfeb
I20260927 02:48:13.870796 8472 ts_tablet_manager.cc:1939] T 846aaa4944b749e180a93adf70e4dfeb P e2e20d67cfe64284b5039cde3c098666: Deleting consensus metadata
I20260927 02:48:13.870863 8451 tablet_bootstrap.cc:492] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Bootstrap starting.
I20260927 02:48:13.871260 8451 tablet_bootstrap.cc:654] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:13.871271 7928 catalog_manager.cc:4841] TS e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539): tablet 846aaa4944b749e180a93adf70e4dfeb (table test-table [id=5442674d17ca41c99bbf0643252dfe9e]) successfully deleted
I20260927 02:48:13.871308 8473 ts_tablet_manager.cc:1918] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:13.871371 8473 log.cc:1196] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/846aaa4944b749e180a93adf70e4dfeb
I20260927 02:48:13.871618 8473 ts_tablet_manager.cc:1939] T 846aaa4944b749e180a93adf70e4dfeb P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting consensus metadata
I20260927 02:48:13.871738 8449 tablet_bootstrap.cc:492] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: No bootstrap required, opened a new log
I20260927 02:48:13.871776 8449 ts_tablet_manager.cc:1397] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 02:48:13.871906 8449 raft_consensus.cc:348] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.871969 8449 raft_consensus.cc:374] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.871989 8449 raft_consensus.cc:729] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.871970 7928 catalog_manager.cc:4841] TS 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109): tablet 846aaa4944b749e180a93adf70e4dfeb (table test-table [id=5442674d17ca41c99bbf0643252dfe9e]) successfully deleted
I20260927 02:48:13.872030 8449 consensus_queue.cc:260] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.872113 8449 ts_tablet_manager.cc:1428] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.872521 8451 tablet_bootstrap.cc:492] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: No bootstrap required, opened a new log
I20260927 02:48:13.872570 8451 ts_tablet_manager.cc:1397] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:13.872702 8451 raft_consensus.cc:348] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.872769 8451 raft_consensus.cc:374] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.872794 8451 raft_consensus.cc:729] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863f3b4e4322499d81e8d4a6d1afa8e7, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.872838 8451 consensus_queue.cc:260] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.872925 8451 ts_tablet_manager.cc:1428] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:13.875231 8452 tablet_bootstrap.cc:492] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: No bootstrap required, opened a new log
I20260927 02:48:13.875281 8452 ts_tablet_manager.cc:1397] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 02:48:13.875424 8452 raft_consensus.cc:348] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.875504 8452 raft_consensus.cc:374] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:13.875524 8452 raft_consensus.cc:729] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c352abe2f4c942f2acd4cd38bd8c3b2e, State: Initialized, Role: FOLLOWER
I20260927 02:48:13.875567 8452 consensus_queue.cc:260] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:13.875653 8452 ts_tablet_manager.cc:1428] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260927 02:48:13.881139 8285 tablet.cc:2369] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:48:13.982838 8145 tablet.cc:2369] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:48:14.041091 8425 tablet.cc:2369] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:14.186234 8455 raft_consensus.cc:482] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:14.186331 8455 raft_consensus.cc:504] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.186656 8455 leader_election.cc:288] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.189555 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.187114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60887 (local address 127.0.98.3:42109)
0927 02:48:14.187167 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.187169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.187178 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:14.187223 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.187224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.187232 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.187278 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:14.187405 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.187853 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.188861 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.189264 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.189323 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.189334 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.189456 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.189486 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:48:14.189581 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.186902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33021 (local address 127.0.98.2:39471)
0927 02:48:14.186955 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.186957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.187096 (+ 139us) server_negotiation.cc:407] Connection header received
0927 02:48:14.187129 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.187130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.187139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.187202 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:14.187365 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.187843 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.188859 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.189280 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.189342 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.189351 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.189485 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.189518 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:48:14.189605 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.187030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42109 (local address 127.0.98.1:60887)
0927 02:48:14.187123 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:14.187132 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:14.187174 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:14.187288 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:14.187289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:14.187291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:14.187349 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.187350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.187923 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.187924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.188774 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.188776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.189331 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.189332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.189397 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.189400 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:14.189445 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:48:14.189486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":44,"thread_start_us":40,"threads_started":1}
I20260927 02:48:14.189615 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.186860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39471 (local address 127.0.98.1:33021)
0927 02:48:14.186967 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:14.186979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:14.187045 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:14.187235 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:14.187236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:14.187239 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:14.187304 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.187309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.187919 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.187920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.188772 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.188774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.189362 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.189363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.189425 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.189428 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:14.189471 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:48:14.189511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":30,"thread_start_us":57,"threads_started":1}
I20260927 02:48:14.189872 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493fe09f69740cc945f34f9a942f072" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" is_pre_election: true
I20260927 02:48:14.189998 8238 raft_consensus.cc:2454] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 0.
I20260927 02:48:14.190232 8032 leader_election.cc:302] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c352abe2f4c942f2acd4cd38bd8c3b2e, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:14.190246 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493fe09f69740cc945f34f9a942f072" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" is_pre_election: true
I20260927 02:48:14.190339 8378 raft_consensus.cc:2454] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 0.
I20260927 02:48:14.190361 8455 raft_consensus.cc:2781] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:14.190392 8455 raft_consensus.cc:482] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:14.190418 8455 raft_consensus.cc:3037] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.191124 8455 raft_consensus.cc:504] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.191234 8455 leader_election.cc:288] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 election: Requested vote from peers c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.191394 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493fe09f69740cc945f34f9a942f072" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e"
I20260927 02:48:14.191463 8238 raft_consensus.cc:3037] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.191418 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9493fe09f69740cc945f34f9a942f072" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
I20260927 02:48:14.191489 8378 raft_consensus.cc:3037] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.192065 8378 raft_consensus.cc:2454] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 1.
I20260927 02:48:14.192075 8238 raft_consensus.cc:2454] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 1.
I20260927 02:48:14.192224 8032 leader_election.cc:302] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863f3b4e4322499d81e8d4a6d1afa8e7, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:14.192303 8455 raft_consensus.cc:2781] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:14.192402 8455 raft_consensus.cc:686] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 1 LEADER]: Becoming Leader. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Running, Role: LEADER
I20260927 02:48:14.192461 8455 consensus_queue.cc:237] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.193097 7941 catalog_manager.cc:5495] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2e20d67cfe64284b5039cde3c098666 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2e20d67cfe64284b5039cde3c098666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477293, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:48:14.250051 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.243501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34067 (local address 127.0.0.1:45138)
0927 02:48:14.243663 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:14.243673 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:14.243742 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:14.243858 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:14.243859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:14.244170 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:14.244215 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.244217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.244827 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.244828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.245340 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.245342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.245645 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.245646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.245691 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.246219 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:14.246224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:14.247544 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:14.249078 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:14.249085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:14.249091 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:14.249292 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:14.249486 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:14.249487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:14.249488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:14.249554 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:14.249772 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:14.249778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:14.249826 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:48:14.249885 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 02:48:14.249976 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260927 02:48:14.250120 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.243599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.0.98.62:34067)
0927 02:48:14.243659 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.243661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.243733 (+ 72us) server_negotiation.cc:407] Connection header received
0927 02:48:14.243794 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.243795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.243804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.243850 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:14.244289 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.244763 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.245406 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.245585 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.245641 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.247630 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:14.247631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:14.247631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:14.247638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:14.248992 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:14.249358 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:14.249359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:14.249359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:14.249410 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:14.249608 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:14.249608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:14.249608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:14.249706 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:14.249751 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.249914 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.250031 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":43}
I20260927 02:48:14.252027 7941 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45138:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 02:48:14.252207 7941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:14.253536 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcfb124c09541749bf486801b598565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc1bb63e728543d095abd4f26116dc12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:48:14.253494 8078 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcfb124c09541749bf486801b598565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc1bb63e728543d095abd4f26116dc12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:48:14.253556 8354 tablet_service.cc:1467] Processing CreateTablet for tablet 0fcfb124c09541749bf486801b598565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc1bb63e728543d095abd4f26116dc12]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 02:48:14.253686 8354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcfb124c09541749bf486801b598565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.253686 8078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcfb124c09541749bf486801b598565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.253687 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fcfb124c09541749bf486801b598565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.254868 8452 tablet_bootstrap.cc:492] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: Bootstrap starting.
I20260927 02:48:14.255124 8449 tablet_bootstrap.cc:492] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: Bootstrap starting.
I20260927 02:48:14.255128 8451 tablet_bootstrap.cc:492] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: Bootstrap starting.
I20260927 02:48:14.255259 8452 tablet_bootstrap.cc:654] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.255455 8451 tablet_bootstrap.cc:654] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.255505 8449 tablet_bootstrap.cc:654] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.255843 8452 tablet_bootstrap.cc:492] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: No bootstrap required, opened a new log
I20260927 02:48:14.255888 8452 ts_tablet_manager.cc:1397] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:14.256002 8452 raft_consensus.cc:348] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.256054 8451 tablet_bootstrap.cc:492] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: No bootstrap required, opened a new log
I20260927 02:48:14.256069 8452 raft_consensus.cc:374] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.256088 8452 raft_consensus.cc:729] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c352abe2f4c942f2acd4cd38bd8c3b2e, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.256091 8451 ts_tablet_manager.cc:1397] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:14.256134 8452 consensus_queue.cc:260] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.256223 8452 ts_tablet_manager.cc:1428] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:14.256225 8449 tablet_bootstrap.cc:492] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: No bootstrap required, opened a new log
I20260927 02:48:14.256210 8451 raft_consensus.cc:348] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.256266 8451 raft_consensus.cc:374] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.256268 8449 ts_tablet_manager.cc:1397] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:48:14.256287 8451 raft_consensus.cc:729] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863f3b4e4322499d81e8d4a6d1afa8e7, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.256243 8457 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.256322 8451 consensus_queue.cc:260] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.256325 8457 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:48:14.256393 8451 ts_tablet_manager.cc:1428] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:14.256403 8449 raft_consensus.cc:348] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.256489 8449 raft_consensus.cc:374] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.256513 8449 raft_consensus.cc:729] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.256546 8449 consensus_queue.cc:260] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.256534 8456 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.256641 8449 ts_tablet_manager.cc:1428] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:14.256649 8456 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:48:14.256783 8455 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.256852 8455 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:48:14.335502 8457 raft_consensus.cc:482] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:14.335631 8457 raft_consensus.cc:504] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.335783 8457 leader_election.cc:288] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.336030 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcfb124c09541749bf486801b598565" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e20d67cfe64284b5039cde3c098666" is_pre_election: true
I20260927 02:48:14.336061 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcfb124c09541749bf486801b598565" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" is_pre_election: true
I20260927 02:48:14.336165 8378 raft_consensus.cc:2454] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 0.
I20260927 02:48:14.336166 8098 raft_consensus.cc:2454] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 0.
I20260927 02:48:14.336328 8172 leader_election.cc:302] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863f3b4e4322499d81e8d4a6d1afa8e7, c352abe2f4c942f2acd4cd38bd8c3b2e; no voters:
I20260927 02:48:14.336419 8457 raft_consensus.cc:2781] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:14.336468 8457 raft_consensus.cc:482] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:14.336488 8457 raft_consensus.cc:3037] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.337128 8457 raft_consensus.cc:504] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.337234 8457 leader_election.cc:288] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 election: Requested vote from peers e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.337409 8098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcfb124c09541749bf486801b598565" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e20d67cfe64284b5039cde3c098666"
I20260927 02:48:14.337423 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fcfb124c09541749bf486801b598565" candidate_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
I20260927 02:48:14.337472 8098 raft_consensus.cc:3037] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.337482 8378 raft_consensus.cc:3037] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.338014 8098 raft_consensus.cc:2454] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 1.
I20260927 02:48:14.338057 8378 raft_consensus.cc:2454] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c352abe2f4c942f2acd4cd38bd8c3b2e in term 1.
I20260927 02:48:14.338176 8172 leader_election.cc:302] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c352abe2f4c942f2acd4cd38bd8c3b2e, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:14.338258 8457 raft_consensus.cc:2781] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:14.338316 8457 raft_consensus.cc:686] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 LEADER]: Becoming Leader. State: Replica: c352abe2f4c942f2acd4cd38bd8c3b2e, State: Running, Role: LEADER
I20260927 02:48:14.338366 8457 consensus_queue.cc:237] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.338611 8459 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: TxnStatusTablet state changed. Reason: New leader c352abe2f4c942f2acd4cd38bd8c3b2e. Latest consensus state: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.338680 8459 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:48:14.338826 8491 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 02:48:14.338940 8491 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 02:48:14.339002 7939 catalog_manager.cc:5495] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260927 02:48:14.351805 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.349222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.0.98.2:39471)
0927 02:48:14.349261 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.349263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.349290 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:48:14.349378 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.349378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.349389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.349438 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:14.349572 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.350047 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.351132 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.351320 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.351372 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.351558 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.351611 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.351659 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.351728 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.351746 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:48:14.351855 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.349165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39471 (local address 127.0.0.1:37610)
0927 02:48:14.349237 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:14.349251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:14.349309 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:14.349454 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:14.349455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:14.349456 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:48:14.349508 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.349511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.350107 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.350108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.351057 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.351059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.351382 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.351383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.351446 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.351461 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.351670 (+ 209us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.351677 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 02:48:14.351724 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:48:14.351775 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:48:14.352756 8098 raft_consensus.cc:1261] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Refusing update from remote peer c352abe2f4c942f2acd4cd38bd8c3b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.352799 8378 raft_consensus.cc:1261] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Refusing update from remote peer c352abe2f4c942f2acd4cd38bd8c3b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.352968 8457 consensus_queue.cc:1035] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.353080 8457 consensus_queue.cc:1035] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.353511 8456 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: TxnStatusTablet state changed. Reason: New leader c352abe2f4c942f2acd4cd38bd8c3b2e. Latest consensus state: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.353513 8455 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: TxnStatusTablet state changed. Reason: New leader c352abe2f4c942f2acd4cd38bd8c3b2e. Latest consensus state: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.353593 8456 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:48:14.353598 8455 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:48:14.354171 8457 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.354243 8457 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:48:14.354213 8462 mvcc.cc:205] Tried to move back new op lower bound from 7333794997667315712 to 7333794997610278912. Current Snapshot: MvccSnapshot[applied={T|T < 7333794997667315712}]
I20260927 02:48:14.354369 8457 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.354422 8457 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P c352abe2f4c942f2acd4cd38bd8c3b2e: This TxnStatusTablet replica's current role is: LEADER
I20260927 02:48:14.354768 8455 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.354789 8456 tablet_replica.cc:439] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } } }
I20260927 02:48:14.354868 8455 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P e2e20d67cfe64284b5039cde3c098666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:48:14.354888 8456 tablet_replica.cc:442] T 0fcfb124c09541749bf486801b598565 P 863f3b4e4322499d81e8d4a6d1afa8e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 02:48:14.358371 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.355724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.0.98.1:41539)
0927 02:48:14.355855 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.355857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.355864 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:48:14.355895 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.355896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.355913 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.355984 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:14.356233 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.356721 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.357756 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.357933 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.357974 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.358111 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.358204 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.358249 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.358306 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.358319 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260927 02:48:14.363051 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.360592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43341 (local address 127.0.98.2:39471)
0927 02:48:14.360640 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.360642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.360758 (+ 116us) server_negotiation.cc:407] Connection header received
0927 02:48:14.360818 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.360818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.360828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.360869 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:14.360982 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.361404 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.362477 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.362646 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.362686 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.362883 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.362905 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.362938 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.362987 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.363002 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:48:14.363123 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.360536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39471 (local address 127.0.98.1:43341)
0927 02:48:14.360701 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:14.360711 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:14.360765 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:14.360873 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:14.360874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:14.360876 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:48:14.360929 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.360931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.361462 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.361462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.362413 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.362415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.362708 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.362708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.362776 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.362790 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.362940 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.362941 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 02:48:14.362984 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:48:14.363024 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260927 02:48:14.365074 8378 raft_consensus.cc:1261] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Refusing update from remote peer e2e20d67cfe64284b5039cde3c098666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.365167 8238 raft_consensus.cc:1261] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Refusing update from remote peer e2e20d67cfe64284b5039cde3c098666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.365293 8455 consensus_queue.cc:1035] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.365401 8479 consensus_queue.cc:1035] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.372931 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.370343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53243 (local address 127.0.98.1:41539)
0927 02:48:14.370385 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.370387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.370500 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:48:14.370574 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.370575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.370589 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.370625 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:14.370734 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.371237 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.372294 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.372554 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.372613 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.372759 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.372784 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.372815 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.372864 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.372880 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:48:14.372982 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.370309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41539 (local address 127.0.98.2:53243)
0927 02:48:14.370448 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:14.370459 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:14.370513 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:14.370639 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:14.370640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:14.370641 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 02:48:14.370686 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.370687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.371297 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.371298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.372219 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:14.372221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.372621 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.372622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:14.372679 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.372693 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.372817 (+ 124us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.372819 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:48:14.372862 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:48:14.372902 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":79,"threads_started":1}
I20260927 02:48:14.381390 7940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:14.381482 7940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:14.382018 7940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Sending DeleteTablet for 3 replicas of tablet 9493fe09f69740cc945f34f9a942f072
I20260927 02:48:14.382336 8078 tablet_service.cc:1514] Processing DeleteTablet for tablet 9493fe09f69740cc945f34f9a942f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35452
I20260927 02:48:14.382329 8354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9493fe09f69740cc945f34f9a942f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39988
I20260927 02:48:14.382550 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet 9493fe09f69740cc945f34f9a942f072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37584
I20260927 02:48:14.382709 8509 tablet_replica.cc:330] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: stopping tablet replica
I20260927 02:48:14.382758 8509 raft_consensus.cc:2229] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:14.382870 8509 raft_consensus.cc:2258] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.384982 8509 ts_tablet_manager.cc:1905] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.385367 8510 tablet_replica.cc:330] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: stopping tablet replica
I20260927 02:48:14.385457 8510 raft_consensus.cc:2229] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:14.385540 8510 raft_consensus.cc:2258] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.385767 8510 ts_tablet_manager.cc:1905] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.387316 8510 ts_tablet_manager.cc:1918] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:48:14.387420 8510 log.cc:1196] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/9493fe09f69740cc945f34f9a942f072
I20260927 02:48:14.387699 8510 ts_tablet_manager.cc:1939] T 9493fe09f69740cc945f34f9a942f072 P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting consensus metadata
I20260927 02:48:14.388098 7928 catalog_manager.cc:4841] TS c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471): tablet 9493fe09f69740cc945f34f9a942f072 (table test-table [id=00acb60a9bed49a888ffd6799e666b46]) successfully deleted
I20260927 02:48:14.388154 8509 ts_tablet_manager.cc:1918] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:48:14.388195 8509 log.cc:1196] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/9493fe09f69740cc945f34f9a942f072
I20260927 02:48:14.388379 8509 ts_tablet_manager.cc:1939] T 9493fe09f69740cc945f34f9a942f072 P e2e20d67cfe64284b5039cde3c098666: Deleting consensus metadata
I20260927 02:48:14.388674 7928 catalog_manager.cc:4841] TS e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539): tablet 9493fe09f69740cc945f34f9a942f072 (table test-table [id=00acb60a9bed49a888ffd6799e666b46]) successfully deleted
I20260927 02:48:14.394884 8508 tablet_replica.cc:330] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: stopping tablet replica
I20260927 02:48:14.394968 8508 raft_consensus.cc:2229] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:14.395063 8508 raft_consensus.cc:2258] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[7896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260927 02:48:14.395505 8508 ts_tablet_manager.cc:1905] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.397143 8508 ts_tablet_manager.cc:1918] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:48:14.397238 8508 log.cc:1196] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/9493fe09f69740cc945f34f9a942f072
I20260927 02:48:14.397599 8508 ts_tablet_manager.cc:1939] T 9493fe09f69740cc945f34f9a942f072 P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting consensus metadata
I20260927 02:48:14.397961 7928 catalog_manager.cc:4841] TS 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109): tablet 9493fe09f69740cc945f34f9a942f072 (table test-table [id=00acb60a9bed49a888ffd6799e666b46]) successfully deleted
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:48:14.412643 7990 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:14.401540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.0.98.62:34067)
0927 02:48:14.401585 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.401587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.401754 (+ 167us) server_negotiation.cc:407] Connection header received
0927 02:48:14.401855 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.401856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.401866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.401946 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:14.402655 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.403437 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.404528 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.404830 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.404864 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.412578 (+ 7714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45146: BlockingRecv error: recv got EOF from 127.0.0.1:45146 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[7896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 02:48:14.421022 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.413626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45156 (local address 127.0.98.62:34067)
0927 02:48:14.413665 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.413667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.414992 (+ 1325us) server_negotiation.cc:407] Connection header received
0927 02:48:14.415049 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.415050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.415061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.415112 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:14.415528 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.415998 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.416653 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.416831 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.416883 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.418857 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:14.418859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:14.418859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:14.418866 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:14.420069 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:14.420360 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:14.420361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:14.420361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:14.420412 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:14.420569 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:14.420570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:14.420570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:14.420680 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:14.420763 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.420861 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.420928 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:48:14.422407 7940 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:14.422555 7940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:14.423799 8078 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3863d3a77449b6a0f6292122d8790e (DEFAULT_TABLE table=test-table [id=8942c483890b4d7e883da21f64203543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:14.423930 8078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3863d3a77449b6a0f6292122d8790e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.423892 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3863d3a77449b6a0f6292122d8790e (DEFAULT_TABLE table=test-table [id=8942c483890b4d7e883da21f64203543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:14.424002 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3863d3a77449b6a0f6292122d8790e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.425242 8449 tablet_bootstrap.cc:492] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: Bootstrap starting.
I20260927 02:48:14.425276 8354 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3863d3a77449b6a0f6292122d8790e (DEFAULT_TABLE table=test-table [id=8942c483890b4d7e883da21f64203543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:14.425374 8354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3863d3a77449b6a0f6292122d8790e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.425616 8449 tablet_bootstrap.cc:654] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.426556 8449 tablet_bootstrap.cc:492] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: No bootstrap required, opened a new log
I20260927 02:48:14.426596 8449 ts_tablet_manager.cc:1397] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:48:14.426661 8452 tablet_bootstrap.cc:492] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: Bootstrap starting.
I20260927 02:48:14.426733 8449 raft_consensus.cc:348] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.426798 8449 raft_consensus.cc:374] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.426914 8449 raft_consensus.cc:729] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.426957 8449 consensus_queue.cc:260] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.426978 8452 tablet_bootstrap.cc:654] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.427048 8449 ts_tablet_manager.cc:1428] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:14.427301 8451 tablet_bootstrap.cc:492] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: Bootstrap starting.
I20260927 02:48:14.427597 8451 tablet_bootstrap.cc:654] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.428442 8451 tablet_bootstrap.cc:492] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: No bootstrap required, opened a new log
I20260927 02:48:14.428484 8451 ts_tablet_manager.cc:1397] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:14.428617 8451 raft_consensus.cc:348] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.428673 8451 raft_consensus.cc:374] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.428689 8451 raft_consensus.cc:729] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863f3b4e4322499d81e8d4a6d1afa8e7, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.428725 8451 consensus_queue.cc:260] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.428807 8451 ts_tablet_manager.cc:1428] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:14.429481 8452 tablet_bootstrap.cc:492] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: No bootstrap required, opened a new log
I20260927 02:48:14.429524 8452 ts_tablet_manager.cc:1397] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 02:48:14.429651 8452 raft_consensus.cc:348] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.429706 8452 raft_consensus.cc:374] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.429723 8452 raft_consensus.cc:729] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c352abe2f4c942f2acd4cd38bd8c3b2e, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.429759 8452 consensus_queue.cc:260] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.429838 8452 ts_tablet_manager.cc:1428] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:14.509663 8506 raft_consensus.cc:482] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:14.509761 8506 raft_consensus.cc:504] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.509912 8506 leader_election.cc:288] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.510125 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3863d3a77449b6a0f6292122d8790e" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" is_pre_election: true
I20260927 02:48:14.510170 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3863d3a77449b6a0f6292122d8790e" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" is_pre_election: true
I20260927 02:48:14.510221 8238 raft_consensus.cc:2454] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 0.
I20260927 02:48:14.510243 8378 raft_consensus.cc:2454] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 0.
I20260927 02:48:14.510381 8032 leader_election.cc:302] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863f3b4e4322499d81e8d4a6d1afa8e7, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:14.510488 8506 raft_consensus.cc:2781] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:14.510558 8506 raft_consensus.cc:482] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:14.510581 8506 raft_consensus.cc:3037] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.511497 8506 raft_consensus.cc:504] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.511642 8506 leader_election.cc:288] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 election: Requested vote from peers c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.511822 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3863d3a77449b6a0f6292122d8790e" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
I20260927 02:48:14.511925 8378 raft_consensus.cc:3037] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.511974 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3863d3a77449b6a0f6292122d8790e" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e"
I20260927 02:48:14.512071 8238 raft_consensus.cc:3037] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.512629 8378 raft_consensus.cc:2454] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 1.
I20260927 02:48:14.512801 8238 raft_consensus.cc:2454] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 1.
I20260927 02:48:14.512833 8032 leader_election.cc:302] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863f3b4e4322499d81e8d4a6d1afa8e7, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:14.513028 8506 raft_consensus.cc:2781] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:14.513095 8506 raft_consensus.cc:686] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 1 LEADER]: Becoming Leader. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Running, Role: LEADER
I20260927 02:48:14.513146 8506 consensus_queue.cc:237] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.513778 7939 catalog_manager.cc:5495] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2e20d67cfe64284b5039cde3c098666 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2e20d67cfe64284b5039cde3c098666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260927 02:48:14.519048 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:14.516965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.0.98.1:41539)
0927 02:48:14.517016 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:14.517018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:14.517024 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:48:14.517046 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:14.517047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:14.517056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:14.517100 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:14.517225 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.517712 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.518426 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:14.518632 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:14.518685 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:14.518830 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:14.518865 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:14.518908 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:14.518962 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 02:48:14.518982 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:48:14.519692 8378 raft_consensus.cc:1261] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Refusing update from remote peer e2e20d67cfe64284b5039cde3c098666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.519824 8238 raft_consensus.cc:1261] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Refusing update from remote peer e2e20d67cfe64284b5039cde3c098666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.519901 8506 consensus_queue.cc:1035] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.520005 8455 consensus_queue.cc:1035] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.522476 7941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:14.522557 7941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:14.523197 7941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Sending DeleteTablet for 3 replicas of tablet 0b3863d3a77449b6a0f6292122d8790e
I20260927 02:48:14.523437 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3863d3a77449b6a0f6292122d8790e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37584
I20260927 02:48:14.523519 8510 tablet_replica.cc:330] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: stopping tablet replica
I20260927 02:48:14.523557 8510 raft_consensus.cc:2229] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:14.523552 8078 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3863d3a77449b6a0f6292122d8790e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35452
I20260927 02:48:14.523602 8510 raft_consensus.cc:2258] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.523644 8077 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3863d3a77449b6a0f6292122d8790e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35452
I20260927 02:48:14.523716 8354 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3863d3a77449b6a0f6292122d8790e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39988
I20260927 02:48:14.523789 8509 tablet_replica.cc:330] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: stopping tablet replica
I20260927 02:48:14.523834 8509 raft_consensus.cc:2229] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:14.523880 8509 raft_consensus.cc:2258] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.523940 8508 tablet_replica.cc:330] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: stopping tablet replica
I20260927 02:48:14.523989 8508 raft_consensus.cc:2229] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:14.524049 8508 raft_consensus.cc:2258] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.524286 8510 ts_tablet_manager.cc:1905] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.524362 7941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 02:48:14.524425 8509 ts_tablet_manager.cc:1905] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:48:14.524484 7941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:14.525770 8508 ts_tablet_manager.cc:1905] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.525909 8509 ts_tablet_manager.cc:1918] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:14.525959 8509 log.cc:1196] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/0b3863d3a77449b6a0f6292122d8790e
I20260927 02:48:14.526140 8509 ts_tablet_manager.cc:1939] T 0b3863d3a77449b6a0f6292122d8790e P e2e20d67cfe64284b5039cde3c098666: Deleting consensus metadata
I20260927 02:48:14.526511 7928 catalog_manager.cc:4841] TS e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539): tablet 0b3863d3a77449b6a0f6292122d8790e (table test-table [id=8942c483890b4d7e883da21f64203543]) successfully deleted
I20260927 02:48:14.526594 8354 tablet_service.cc:1467] Processing CreateTablet for tablet 76987fba5acd498a86e01afa337e96aa (DEFAULT_TABLE table=test-table [id=0b3214a881d245eb8d712a4228bc984f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260927 02:48:14.526615 7928 catalog_manager.cc:4808] TS e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539): delete failed for tablet 0b3863d3a77449b6a0f6292122d8790e because the tablet was not found. No further retry: Not found: Tablet not found: 0b3863d3a77449b6a0f6292122d8790e
I20260927 02:48:14.526679 8354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76987fba5acd498a86e01afa337e96aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.526772 8218 tablet_service.cc:1467] Processing CreateTablet for tablet 76987fba5acd498a86e01afa337e96aa (DEFAULT_TABLE table=test-table [id=0b3214a881d245eb8d712a4228bc984f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:14.526881 8077 tablet_service.cc:1467] Processing CreateTablet for tablet 76987fba5acd498a86e01afa337e96aa (DEFAULT_TABLE table=test-table [id=0b3214a881d245eb8d712a4228bc984f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:14.526968 8077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76987fba5acd498a86e01afa337e96aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.526968 8510 ts_tablet_manager.cc:1918] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:14.527006 8510 log.cc:1196] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/0b3863d3a77449b6a0f6292122d8790e
I20260927 02:48:14.527170 8508 ts_tablet_manager.cc:1918] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:14.527200 8510 ts_tablet_manager.cc:1939] T 0b3863d3a77449b6a0f6292122d8790e P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting consensus metadata
I20260927 02:48:14.527213 8508 log.cc:1196] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/0b3863d3a77449b6a0f6292122d8790e
I20260927 02:48:14.527334 8218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76987fba5acd498a86e01afa337e96aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:14.527396 8508 ts_tablet_manager.cc:1939] T 0b3863d3a77449b6a0f6292122d8790e P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting consensus metadata
I20260927 02:48:14.528170 8449 tablet_bootstrap.cc:492] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: Bootstrap starting.
I20260927 02:48:14.528551 8449 tablet_bootstrap.cc:654] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.528573 8452 tablet_bootstrap.cc:492] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: Bootstrap starting.
I20260927 02:48:14.528824 8452 tablet_bootstrap.cc:654] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.529134 8449 tablet_bootstrap.cc:492] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: No bootstrap required, opened a new log
I20260927 02:48:14.529170 8449 ts_tablet_manager.cc:1397] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 02:48:14.529294 8449 raft_consensus.cc:348] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.529349 8449 raft_consensus.cc:374] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.529367 8449 raft_consensus.cc:729] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.529405 8449 consensus_queue.cc:260] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.529490 8449 ts_tablet_manager.cc:1428] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:14.529947 8451 tablet_bootstrap.cc:492] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: Bootstrap starting.
I20260927 02:48:14.530220 8452 tablet_bootstrap.cc:492] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: No bootstrap required, opened a new log
I20260927 02:48:14.530267 8452 ts_tablet_manager.cc:1397] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:48:14.530359 8452 raft_consensus.cc:348] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.530411 8452 raft_consensus.cc:374] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.530428 8452 raft_consensus.cc:729] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c352abe2f4c942f2acd4cd38bd8c3b2e, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.530463 8452 consensus_queue.cc:260] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.530500 8451 tablet_bootstrap.cc:654] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:14.530562 8452 ts_tablet_manager.cc:1428] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:14.530978 7928 catalog_manager.cc:4841] TS c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471): tablet 0b3863d3a77449b6a0f6292122d8790e (table test-table [id=8942c483890b4d7e883da21f64203543]) successfully deleted
I20260927 02:48:14.531049 7928 catalog_manager.cc:4841] TS 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109): tablet 0b3863d3a77449b6a0f6292122d8790e (table test-table [id=8942c483890b4d7e883da21f64203543]) successfully deleted
I20260927 02:48:14.531538 8451 tablet_bootstrap.cc:492] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: No bootstrap required, opened a new log
I20260927 02:48:14.531581 8451 ts_tablet_manager.cc:1397] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260927 02:48:14.531718 8451 raft_consensus.cc:348] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.531762 8451 raft_consensus.cc:374] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:14.531773 8451 raft_consensus.cc:729] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863f3b4e4322499d81e8d4a6d1afa8e7, State: Initialized, Role: FOLLOWER
I20260927 02:48:14.531806 8451 consensus_queue.cc:260] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.531878 8451 ts_tablet_manager.cc:1428] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:14.602608 8455 raft_consensus.cc:482] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:14.602710 8455 raft_consensus.cc:504] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.602897 8455 leader_election.cc:288] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.603092 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987fba5acd498a86e01afa337e96aa" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" is_pre_election: true
I20260927 02:48:14.603111 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987fba5acd498a86e01afa337e96aa" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" is_pre_election: true
I20260927 02:48:14.603192 8238 raft_consensus.cc:2454] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 0.
I20260927 02:48:14.603191 8378 raft_consensus.cc:2454] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 0.
I20260927 02:48:14.603341 8032 leader_election.cc:302] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c352abe2f4c942f2acd4cd38bd8c3b2e, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:14.603440 8455 raft_consensus.cc:2781] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:14.603488 8455 raft_consensus.cc:482] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:14.603507 8455 raft_consensus.cc:3037] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.604158 8455 raft_consensus.cc:504] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.604260 8455 leader_election.cc:288] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 election: Requested vote from peers c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471), 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109)
I20260927 02:48:14.604422 8238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987fba5acd498a86e01afa337e96aa" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e"
I20260927 02:48:14.604499 8238 raft_consensus.cc:3037] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.604468 8378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76987fba5acd498a86e01afa337e96aa" candidate_uuid: "e2e20d67cfe64284b5039cde3c098666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7"
I20260927 02:48:14.604537 8378 raft_consensus.cc:3037] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:14.605108 8378 raft_consensus.cc:2454] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 1.
I20260927 02:48:14.605144 8238 raft_consensus.cc:2454] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2e20d67cfe64284b5039cde3c098666 in term 1.
I20260927 02:48:14.605273 8032 leader_election.cc:302] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 863f3b4e4322499d81e8d4a6d1afa8e7, e2e20d67cfe64284b5039cde3c098666; no voters:
I20260927 02:48:14.605371 8455 raft_consensus.cc:2781] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:14.605429 8455 raft_consensus.cc:686] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 1 LEADER]: Becoming Leader. State: Replica: e2e20d67cfe64284b5039cde3c098666, State: Running, Role: LEADER
I20260927 02:48:14.605472 8455 consensus_queue.cc:237] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } }
I20260927 02:48:14.606017 7941 catalog_manager.cc:5495] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2e20d67cfe64284b5039cde3c098666 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2e20d67cfe64284b5039cde3c098666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2e20d67cfe64284b5039cde3c098666" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 } health_report { overall_health: UNKNOWN } } }
I20260927 02:48:14.626350 8238 raft_consensus.cc:1261] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Refusing update from remote peer e2e20d67cfe64284b5039cde3c098666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.626349 8378 raft_consensus.cc:1261] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Refusing update from remote peer e2e20d67cfe64284b5039cde3c098666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:14.626561 8506 consensus_queue.cc:1035] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "863f3b4e4322499d81e8d4a6d1afa8e7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.626662 8455 consensus_queue.cc:1035] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c352abe2f4c942f2acd4cd38bd8c3b2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:14.636228 7939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:14.636317 7939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:14.636824 7939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67d4dc0acc8d42f1a3b0fd1875f87ee3: Sending DeleteTablet for 3 replicas of tablet 76987fba5acd498a86e01afa337e96aa
I20260927 02:48:14.637027 8077 tablet_service.cc:1514] Processing DeleteTablet for tablet 76987fba5acd498a86e01afa337e96aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35452
I20260927 02:48:14.637045 8218 tablet_service.cc:1514] Processing DeleteTablet for tablet 76987fba5acd498a86e01afa337e96aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37584
I20260927 02:48:14.637161 8510 tablet_replica.cc:330] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: stopping tablet replica
I20260927 02:48:14.637179 8509 tablet_replica.cc:330] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: stopping tablet replica
I20260927 02:48:14.637181 8354 tablet_service.cc:1514] Processing DeleteTablet for tablet 76987fba5acd498a86e01afa337e96aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39988
I20260927 02:48:14.637228 8509 raft_consensus.cc:2229] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:14.637228 8510 raft_consensus.cc:2229] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:14.637257 8508 tablet_replica.cc:330] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: stopping tablet replica
I20260927 02:48:14.637269 8510 raft_consensus.cc:2258] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.637283 8509 raft_consensus.cc:2258] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.637305 8508 raft_consensus.cc:2229] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:14.637339 8508 raft_consensus.cc:2258] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:14.637547 8509 ts_tablet_manager.cc:1905] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.637579 8508 ts_tablet_manager.cc:1905] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.637629 8510 ts_tablet_manager.cc:1905] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:14.638928 8509 ts_tablet_manager.cc:1918] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:48:14.638959 8508 ts_tablet_manager.cc:1918] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:48:14.638973 8510 ts_tablet_manager.cc:1918] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 02:48:14.638989 8509 log.cc:1196] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/76987fba5acd498a86e01afa337e96aa
I20260927 02:48:14.639007 8508 log.cc:1196] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/76987fba5acd498a86e01afa337e96aa
I20260927 02:48:14.639011 8510 log.cc:1196] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/76987fba5acd498a86e01afa337e96aa
I20260927 02:48:14.639210 8510 ts_tablet_manager.cc:1939] T 76987fba5acd498a86e01afa337e96aa P c352abe2f4c942f2acd4cd38bd8c3b2e: Deleting consensus metadata
I20260927 02:48:14.639210 8508 ts_tablet_manager.cc:1939] T 76987fba5acd498a86e01afa337e96aa P 863f3b4e4322499d81e8d4a6d1afa8e7: Deleting consensus metadata
I20260927 02:48:14.639209 8509 ts_tablet_manager.cc:1939] T 76987fba5acd498a86e01afa337e96aa P e2e20d67cfe64284b5039cde3c098666: Deleting consensus metadata
I20260927 02:48:14.639511 7928 catalog_manager.cc:4841] TS 863f3b4e4322499d81e8d4a6d1afa8e7 (127.0.98.3:42109): tablet 76987fba5acd498a86e01afa337e96aa (table test-table [id=0b3214a881d245eb8d712a4228bc984f]) successfully deleted
I20260927 02:48:14.639588 7928 catalog_manager.cc:4841] TS c352abe2f4c942f2acd4cd38bd8c3b2e (127.0.98.2:39471): tablet 76987fba5acd498a86e01afa337e96aa (table test-table [id=0b3214a881d245eb8d712a4228bc984f]) successfully deleted
I20260927 02:48:14.639618 7928 catalog_manager.cc:4841] TS e2e20d67cfe64284b5039cde3c098666 (127.0.98.1:41539): tablet 76987fba5acd498a86e01afa337e96aa (table test-table [id=0b3214a881d245eb8d712a4228bc984f]) successfully deleted
I20260927 02:48:14.639827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 8013
I20260927 02:48:14.646062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 8153
I20260927 02:48:14.651036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 8293
I20260927 02:48:14.654883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 7909
2026-09-27T02:48:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2533 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[8540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[8540](info): set up 2 sockets
Sep 27 02:48:14 dist-test-slave-f4rg krb5kdc[8540](info): commencing operation
krb5kdc: starting...
W20260927 02:48:15.608469 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:15 dist-test-slave-f4rg krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:15.719879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38183
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:15.797710 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:15.797912 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:15.797932 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:15.799043 8553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:15.799078 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:15.799090 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:15.799101 8553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:15.799111 8553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:15.800534 8553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:15.800729 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:15.800895 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:15.805277 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:15.805362 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:15.805357 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:15.805505 8553 server_base.cc:1034] running on GCE node
I20260927 02:48:15.805717 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:15.805966 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:15.807106 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790477295807092 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T02:48:15Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1152 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:15 dist-test-slave-f4rg krb5kdc[8566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:15 dist-test-slave-f4rg krb5kdc[8566](info): set up 2 sockets
Sep 27 02:48:15 dist-test-slave-f4rg krb5kdc[8566](info): commencing operation
krb5kdc: starting...
W20260927 02:48:16.754993 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:16 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:16.865234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:16.943140 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:16.943339 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:16.943360 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:16.944484 8579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:16.944523 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:16.944536 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:16.944551 8579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:16.944576 8579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:16.946039 8579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:16.946252 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:16.946444 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:16.950893 8587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:16.950966 8585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:16.951061 8579 server_base.cc:1034] running on GCE node
W20260927 02:48:16.951097 8584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:16.951320 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:16.951588 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:16.952735 8579 hybrid_clock.cc:648] HybridClock initialized: now 1790477296952715 us; error 31 us; skew 500 ppm
Sep 27 02:48:16 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477296, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:16.955015 8579 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:16.955428 8579 webserver.cc:460] Webserver started at http://127.0.98.62:41227/ using document root <none> and password file <none>
I20260927 02:48:16.955639 8579 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:16.955685 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:16.955803 8579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:16.956796 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "043f34f3aa094ca9bbec3ab06e5519fa"
format_stamp: "Formatted at 2026-09-27 02:48:16 on dist-test-slave-f4rg"
server_key: "689d422a8f8ce63d08d2b7d5b40bab04"
server_key_iv: "0c7ac60de521e423359b408073d9c268"
server_key_version: "encryptionkey@0"
I20260927 02:48:16.957113 8579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "043f34f3aa094ca9bbec3ab06e5519fa"
format_stamp: "Formatted at 2026-09-27 02:48:16 on dist-test-slave-f4rg"
server_key: "689d422a8f8ce63d08d2b7d5b40bab04"
server_key_iv: "0c7ac60de521e423359b408073d9c268"
server_key_version: "encryptionkey@0"
I20260927 02:48:16.958279 8579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:48:16.958896 8594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:16.959074 8579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:16.959192 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "043f34f3aa094ca9bbec3ab06e5519fa"
format_stamp: "Formatted at 2026-09-27 02:48:16 on dist-test-slave-f4rg"
server_key: "689d422a8f8ce63d08d2b7d5b40bab04"
server_key_iv: "0c7ac60de521e423359b408073d9c268"
server_key_version: "encryptionkey@0"
I20260927 02:48:16.959252 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:16.978427 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:16.980592 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:16.980751 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:16.984488 8579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38509
I20260927 02:48:16.984530 8646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38509 every 8 connection(s)
I20260927 02:48:16.985023 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:16.985647 8647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:16.987879 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Bootstrap starting.
I20260927 02:48:16.988448 8647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:16.988708 8647 log.cc:824] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:16.989317 8647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: No bootstrap required, opened a new log
I20260927 02:48:16.989745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 8579
I20260927 02:48:16.989850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:16.990024 392 external_mini_cluster.cc:1428] Setting key 42b76800a5a6cc1722f89dff9e21812e
I20260927 02:48:16.991683 8647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:16.991851 8647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:16.991887 8647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043f34f3aa094ca9bbec3ab06e5519fa, State: Initialized, Role: FOLLOWER
I20260927 02:48:16.992034 8647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:16.992112 8647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:16.992133 8647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:16.992179 8647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:16.992789 8647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:16.992892 8647 leader_election.cc:302] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043f34f3aa094ca9bbec3ab06e5519fa; no voters:
I20260927 02:48:16.993037 8647 leader_election.cc:288] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:16.993080 8652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:16.993271 8647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:16.993290 8652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 LEADER]: Becoming Leader. State: Replica: 043f34f3aa094ca9bbec3ab06e5519fa, State: Running, Role: LEADER
I20260927 02:48:16.993443 8652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:16.993798 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } } }
I20260927 02:48:16.993868 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:16.994076 8654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043f34f3aa094ca9bbec3ab06e5519fa. Latest consensus state: current_term: 1 leader_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } } }
I20260927 02:48:16.994339 8654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: This master's current role is: LEADER
W20260927 02:48:16.994999 8668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:48:16.995041 8668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:48:16.995082 8660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:16.995241 8660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:16.996582 8660 catalog_manager.cc:1347] Generated new cluster ID: 29162e01bc0f4e03bf24815888cb7a84
I20260927 02:48:16.996628 8660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:48:16 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:17.001909 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:16.990517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.0.98.62:38509)
0927 02:48:16.990719 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:16.990730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:16.990764 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:16.991616 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:16.991625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:16.991689 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:16.991839 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:16.992494 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:16.993823 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:16.994923 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:16.995520 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:16.995628 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:16.998244 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:16.998248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:16.998248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:16.998260 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:17.000208 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.000571 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.000574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.000575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.000687 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.000852 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.000852 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.000853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.001178 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:17.001363 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:17.001445 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 02:48:17.001700 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":96,"threads_started":1}
I20260927 02:48:17.007755 8660 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:17.008368 8660 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:17.011355 8660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Generated new TSK 0
I20260927 02:48:17.011538 8660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:48:17.037907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:17.116978 8673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:17.117192 8673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:17.117213 8673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:17.118351 8673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:17.118391 8673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:17.118420 8673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:17.119887 8673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:17.120108 8673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:17.120309 8673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:17.124898 8678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.124895 8681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.124895 8679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.124981 8673 server_base.cc:1034] running on GCE node
I20260927 02:48:17.125295 8673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:17.125500 8673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:17.126655 8673 hybrid_clock.cc:648] HybridClock initialized: now 1790477297126642 us; error 31 us; skew 500 ppm
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:17.128970 8673 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:48:17.129360 8673 webserver.cc:460] Webserver started at http://127.0.98.1:43835/ using document root <none> and password file <none>
I20260927 02:48:17.129554 8673 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:17.129601 8673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:17.129719 8673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:17.130734 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "051c4b9249da409cbd451d7d87a865dd"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "d0f24fd8d1110dd2acb2a2095173db0c"
server_key_iv: "9f1623e74f4670e9f1ad98aa2d75e660"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.131103 8673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "051c4b9249da409cbd451d7d87a865dd"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "d0f24fd8d1110dd2acb2a2095173db0c"
server_key_iv: "9f1623e74f4670e9f1ad98aa2d75e660"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.132319 8673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:17.132933 8688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.133073 8673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:17.133148 8673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "051c4b9249da409cbd451d7d87a865dd"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "d0f24fd8d1110dd2acb2a2095173db0c"
server_key_iv: "9f1623e74f4670e9f1ad98aa2d75e660"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.133205 8673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:17.145593 8673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:17.147341 8673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:17.147472 8673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:17.147743 8673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:17.148108 8673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:17.148149 8673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.148187 8673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:17.148204 8673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.153671 8673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42315
I20260927 02:48:17.153729 8801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42315 every 8 connection(s)
I20260927 02:48:17.154006 8673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:17.162451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 8673
I20260927 02:48:17.162562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:17.162770 392 external_mini_cluster.cc:1428] Setting key fad865f2fb3b27f8869888237b59f126
I20260927 02:48:17.162777 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.154688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40871 (local address 127.0.98.62:38509)
0927 02:48:17.154741 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.154743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.154939 (+ 196us) server_negotiation.cc:407] Connection header received
0927 02:48:17.155839 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.155840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.155849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.155905 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.156474 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.156988 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.157946 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.158147 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.158195 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.160166 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:17.160167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:17.160168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:17.160175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:17.161430 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.161815 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.161817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.161817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.161883 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.162119 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.162120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.162120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.162229 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:17.162282 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:17.162628 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 02:48:17.162709 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:48:17.163017 8804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.154652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:40871)
0927 02:48:17.154791 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.154863 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.155761 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.155962 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.155965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.156215 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.156404 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.156408 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.157057 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.157058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.157872 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.157874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.158205 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.158206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.158299 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.158713 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.158720 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.160060 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:17.161502 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:17.161508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:17.161510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:17.161729 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:17.161946 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:17.161946 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:17.161947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:17.162069 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:17.162283 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:17.162287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:17.162469 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 02:48:17.162606 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 02:48:17.162772 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260927 02:48:17.163555 8802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38509
I20260927 02:48:17.163647 8802 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:17.163815 8802 heartbeater.cc:511] Master 127.0.98.62:38509 requested a full tablet report, sending...
I20260927 02:48:17.164309 8611 ts_manager.cc:194] Registered new tserver with Master: 051c4b9249da409cbd451d7d87a865dd (127.0.98.1:42315)
I20260927 02:48:17.165067 8611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40871
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:48:17.185509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:17.263697 8807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:17.263903 8807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:17.263926 8807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:17.265085 8807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:17.265132 8807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:17.265163 8807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:17.266681 8807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:17.266927 8807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:17.267148 8807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:17.271981 8812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.272025 8813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.272025 8815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.272080 8807 server_base.cc:1034] running on GCE node
I20260927 02:48:17.272424 8807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:17.272671 8807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:17.273804 8807 hybrid_clock.cc:648] HybridClock initialized: now 1790477297273792 us; error 27 us; skew 500 ppm
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:17.280227 8807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:48:17.280736 8807 webserver.cc:460] Webserver started at http://127.0.98.2:38781/ using document root <none> and password file <none>
I20260927 02:48:17.280946 8807 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:17.280994 8807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:17.281118 8807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:17.282145 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "317a3f90d4474c929d9aed7b6462dfa7"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "f3b5d4c3f2e4fbb66f8234694bb2dfe2"
server_key_iv: "d37894cd3c5ba26d74828a5bd61c5b47"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.282462 8807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "317a3f90d4474c929d9aed7b6462dfa7"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "f3b5d4c3f2e4fbb66f8234694bb2dfe2"
server_key_iv: "d37894cd3c5ba26d74828a5bd61c5b47"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.283727 8807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 02:48:17.284370 8822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.284529 8807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:17.284610 8807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "317a3f90d4474c929d9aed7b6462dfa7"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "f3b5d4c3f2e4fbb66f8234694bb2dfe2"
server_key_iv: "d37894cd3c5ba26d74828a5bd61c5b47"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.284667 8807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:17.303843 8807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:17.305562 8807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:17.305697 8807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:17.305918 8807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:17.306198 8807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:17.306231 8807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.306259 8807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:17.306280 8807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.311826 8807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38677
I20260927 02:48:17.311877 8935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38677 every 8 connection(s)
I20260927 02:48:17.312168 8807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:17.320163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 8807
I20260927 02:48:17.320283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:17.320374 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.312845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52821 (local address 127.0.98.62:38509)
0927 02:48:17.312894 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.312896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.313091 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:48:17.313969 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.313970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.313980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.314032 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.314630 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.315124 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.316061 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.316234 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.316277 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.318153 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:17.318154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:17.318155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:17.318161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:17.319293 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.319639 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.319640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.319640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.319683 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.319853 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.319853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.319853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.319944 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:17.319990 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:17.320202 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 02:48:17.320274 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:48:17.320477 392 external_mini_cluster.cc:1428] Setting key d99ffee9d8ced19c45a81e436198f5c8
I20260927 02:48:17.320505 8938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.312819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:52821)
0927 02:48:17.312969 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.313006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.313890 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.314097 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.314101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.314359 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.314566 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.314571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.315179 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.315179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.315998 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.316000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.316287 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.316287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.316379 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.316793 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.316802 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.318073 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:17.319357 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:17.319363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:17.319364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:17.319587 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:17.319725 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:17.319726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:17.319726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:17.319813 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:17.319989 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:17.319991 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:17.320096 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 02:48:17.320167 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:48:17.320299 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260927 02:48:17.320961 8936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38509
I20260927 02:48:17.321065 8936 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:17.321241 8936 heartbeater.cc:511] Master 127.0.98.62:38509 requested a full tablet report, sending...
I20260927 02:48:17.321597 8610 ts_manager.cc:194] Registered new tserver with Master: 317a3f90d4474c929d9aed7b6462dfa7 (127.0.98.2:38677)
I20260927 02:48:17.321959 8610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52821
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:48:17.342799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:17.424808 8941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:17.425021 8941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:17.425053 8941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:17.426190 8941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:17.426227 8941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:17.426260 8941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:17.427766 8941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:17.428004 8941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:17.428206 8941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:17.432857 8947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.432837 8946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.432832 8949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.432963 8941 server_base.cc:1034] running on GCE node
I20260927 02:48:17.433264 8941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:17.433490 8941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:17.434630 8941 hybrid_clock.cc:648] HybridClock initialized: now 1790477297434617 us; error 27 us; skew 500 ppm
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:17.437079 8941 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:48:17.437507 8941 webserver.cc:460] Webserver started at http://127.0.98.3:44043/ using document root <none> and password file <none>
I20260927 02:48:17.437728 8941 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:17.437773 8941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:17.437880 8941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:17.438956 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "d0682b16b06648248ea44e70a70ed4a9"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "ad3c3385c4b92a208abb62cbf08bdeee"
server_key_iv: "2172620dd31968a268c26258c1a66273"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.439277 8941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d0682b16b06648248ea44e70a70ed4a9"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "ad3c3385c4b92a208abb62cbf08bdeee"
server_key_iv: "2172620dd31968a268c26258c1a66273"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.440461 8941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:48:17.441146 8956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.441284 8941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:17.441362 8941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "d0682b16b06648248ea44e70a70ed4a9"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "ad3c3385c4b92a208abb62cbf08bdeee"
server_key_iv: "2172620dd31968a268c26258c1a66273"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.441421 8941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:17.461237 8941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:17.463415 8941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:17.463531 8941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:17.463738 8941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:17.464018 8941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:17.464056 8941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.464083 8941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:17.464105 8941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.469393 8941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41915
I20260927 02:48:17.469435 9069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41915 every 8 connection(s)
I20260927 02:48:17.469749 8941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:17.472026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 8941
I20260927 02:48:17.472136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:17.472322 392 external_mini_cluster.cc:1428] Setting key 871619afee93000aa09148e1daa1f4c4
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:17.478155 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.470414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41021 (local address 127.0.98.62:38509)
0927 02:48:17.470471 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.470473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.470684 (+ 211us) server_negotiation.cc:407] Connection header received
0927 02:48:17.471608 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.471609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.471618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.471672 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.472277 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.472951 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.473933 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.474119 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.474168 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.476123 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:17.476124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:17.476125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:17.476131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:17.477164 (+ 1033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.477485 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.477485 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.477486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.477532 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:17.477710 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:17.477711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:17.477711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:17.477801 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:17.477841 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:17.478038 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 02:48:17.478091 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:48:17.478323 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.470386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:41021)
0927 02:48:17.470570 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.470610 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.471527 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.471728 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.471731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.471978 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.472191 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.472198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.473026 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.473027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.473866 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.473869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.474174 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.474175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.474275 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.474745 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.474754 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.476048 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:17.477224 (+ 1176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:17.477229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:17.477231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:17.477431 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:17.477575 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:17.477575 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:17.477575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:17.477670 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:17.477839 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:17.477842 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:17.477948 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 02:48:17.478020 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:48:17.478135 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260927 02:48:17.478696 9070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38509
I20260927 02:48:17.478786 9070 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:17.478971 9070 heartbeater.cc:511] Master 127.0.98.62:38509 requested a full tablet report, sending...
I20260927 02:48:17.479290 8610 ts_manager.cc:194] Registered new tserver with Master: d0682b16b06648248ea44e70a70ed4a9 (127.0.98.3:41915)
I20260927 02:48:17.479624 8610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41021
I20260927 02:48:17.483806 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 02:48:17.547673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 8673
I20260927 02:48:17.551656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 8807
I20260927 02:48:17.555378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 8941
I20260927 02:48:17.559039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 8579
I20260927 02:48:17.563272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38509
--webserver_interface=127.0.98.62
--webserver_port=41227
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:17.641563 9079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:17.641763 9079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:17.641784 9079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:17.642911 9079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:17.642946 9079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:17.642958 9079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:17.642970 9079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:17.642980 9079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:17.644418 9079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:17.644606 9079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:17.644763 9079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:17.649209 9084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.649214 9087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.649219 9085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.649256 9079 server_base.cc:1034] running on GCE node
I20260927 02:48:17.649658 9079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:17.649938 9079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:17.651108 9079 hybrid_clock.cc:648] HybridClock initialized: now 1790477297651090 us; error 32 us; skew 500 ppm
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:17.653450 9079 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:17.653905 9079 webserver.cc:460] Webserver started at http://127.0.98.62:41227/ using document root <none> and password file <none>
I20260927 02:48:17.654094 9079 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:17.654155 9079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:17.655540 9079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:48:17.656193 9094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.656363 9079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:17.656454 9079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "043f34f3aa094ca9bbec3ab06e5519fa"
format_stamp: "Formatted at 2026-09-27 02:48:16 on dist-test-slave-f4rg"
server_key: "689d422a8f8ce63d08d2b7d5b40bab04"
server_key_iv: "0c7ac60de521e423359b408073d9c268"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.656803 9079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:17.664952 9079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:17.666539 9079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:17.666662 9079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:17.670372 9079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38509
I20260927 02:48:17.670403 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38509 every 8 connection(s)
I20260927 02:48:17.670933 9079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:17.672072 9147 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:48:17.672655 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Bootstrap starting.
I20260927 02:48:17.677359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9079
I20260927 02:48:17.677639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42315
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38509
--webserver_port=43835
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:48:17.680269 9147 log.cc:824] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:17.681828 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:48:17.682072 9147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Bootstrap complete.
I20260927 02:48:17.684137 9147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:17.684362 9147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043f34f3aa094ca9bbec3ab06e5519fa, State: Initialized, Role: FOLLOWER
I20260927 02:48:17.684508 9147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:17.684578 9147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:17.684626 9147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:17.684671 9147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:48:17.685638 9147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:17.685732 9147 leader_election.cc:302] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043f34f3aa094ca9bbec3ab06e5519fa; no voters:
I20260927 02:48:17.685895 9147 leader_election.cc:288] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:48:17.685931 9151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:48:17.686060 9151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [term 2 LEADER]: Becoming Leader. State: Replica: 043f34f3aa094ca9bbec3ab06e5519fa, State: Running, Role: LEADER
I20260927 02:48:17.686172 9147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:17.686203 9151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } }
I20260927 02:48:17.686503 9152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } } }
I20260927 02:48:17.686496 9153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043f34f3aa094ca9bbec3ab06e5519fa. Latest consensus state: current_term: 2 leader_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043f34f3aa094ca9bbec3ab06e5519fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38509 } } }
I20260927 02:48:17.686622 9152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:17.686638 9153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:17.687040 9157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:17.687521 9157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:17.687638 9157 catalog_manager.cc:1259] Loaded cluster ID: 29162e01bc0f4e03bf24815888cb7a84
I20260927 02:48:17.687866 9157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:17.689118 9157 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:17.689280 9157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 043f34f3aa094ca9bbec3ab06e5519fa: Loaded TSK: 0
I20260927 02:48:17.689489 9157 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:48:17.756958 9149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:17.757165 9149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:17.757189 9149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:17.758345 9149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:17.758385 9149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:17.758411 9149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:17.759882 9149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42315
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43835
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:17.760106 9149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:17.760329 9149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:17.765058 9174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.765069 9173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.765167 9149 server_base.cc:1034] running on GCE node
W20260927 02:48:17.765236 9176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.765458 9149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:17.765682 9149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:17.766837 9149 hybrid_clock.cc:648] HybridClock initialized: now 1790477297766820 us; error 30 us; skew 500 ppm
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:17.769129 9149 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 02:48:17.769539 9149 webserver.cc:460] Webserver started at http://127.0.98.1:43835/ using document root <none> and password file <none>
I20260927 02:48:17.769737 9149 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:17.769789 9149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:17.771148 9149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:48:17.771715 9183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.771873 9149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:17.771955 9149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "051c4b9249da409cbd451d7d87a865dd"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "d0f24fd8d1110dd2acb2a2095173db0c"
server_key_iv: "9f1623e74f4670e9f1ad98aa2d75e660"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.772285 9149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:17.790349 9149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:17.792615 9149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:17.792757 9149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:17.792986 9149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:17.793269 9149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:17.793301 9149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.793330 9149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:17.793355 9149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.798743 9149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42315
I20260927 02:48:17.798795 9296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42315 every 8 connection(s)
I20260927 02:48:17.799140 9149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:48:17.802155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9149
I20260927 02:48:17.802413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38677
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38509
--webserver_port=38781
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:17.808224 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.799928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:45065)
0927 02:48:17.800108 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.800156 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.800749 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.801169 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.801173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.801890 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.802155 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.802161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.803774 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.803775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.804826 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.804829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.805384 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.805384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.805513 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.806015 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.806023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.808030 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
W20260927 02:48:17.808634 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:17.800145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45065 (local address 127.0.98.62:38509)
0927 02:48:17.800294 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.800302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.800329 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:48:17.800929 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.800934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.800990 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.801097 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.802284 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.803660 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.804907 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.805308 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.805408 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.808154 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45065: BlockingRecv error: recv got EOF from 127.0.98.1:45065 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
W20260927 02:48:17.808674 9297 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38509: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:17.812879 9299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.808927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:52083)
0927 02:48:17.809035 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.809046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.809084 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.809254 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.809255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.809374 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.809413 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.809414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.809999 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.810000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.810476 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.810477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.810788 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.810789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.810873 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.811282 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.811287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.812833 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93}
W20260927 02:48:17.813000 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:17.808972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52083 (local address 127.0.98.62:38509)
0927 02:48:17.809168 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.809169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.809174 (+ 5us) server_negotiation.cc:407] Connection header received
0927 02:48:17.809190 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.809190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.809198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.809240 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.809482 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.809933 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.810539 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.810735 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.810785 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.812934 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52083: BlockingRecv error: recv got EOF from 127.0.98.1:52083 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:17.817679 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.813086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:42025)
0927 02:48:17.813215 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.813226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.813271 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.813371 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.813372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.813542 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.813601 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.813602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.814139 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.814139 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.815057 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.815058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.815434 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.815435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.815495 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.815943 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.815947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.817595 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260927 02:48:17.817716 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:17.813246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42025 (local address 127.0.98.62:38509)
0927 02:48:17.813281 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.813282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.813290 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:48:17.813323 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.813324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.813332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.813366 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.813655 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.814081 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.815157 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.815341 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.815413 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.817671 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42025: BlockingRecv error: recv got EOF from 127.0.98.1:42025 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:48:17.817796 9297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:48:17.883086 9301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:17.883291 9301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:17.883313 9301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:17.884454 9301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:17.884495 9301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:17.884519 9301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:17.885910 9301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38781
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:17.886121 9301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:17.886318 9301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:17.891141 9309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:17.891237 9311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.891189 9301 server_base.cc:1034] running on GCE node
W20260927 02:48:17.891294 9308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:17.891533 9301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:17.891759 9301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:17.892895 9301 hybrid_clock.cc:648] HybridClock initialized: now 1790477297892886 us; error 25 us; skew 500 ppm
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477297, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:17.895277 9301 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 02:48:17.895702 9301 webserver.cc:460] Webserver started at http://127.0.98.2:38781/ using document root <none> and password file <none>
I20260927 02:48:17.895900 9301 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:17.895959 9301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:17.897228 9301 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:17.897691 9318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.897835 9301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.897913 9301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "317a3f90d4474c929d9aed7b6462dfa7"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "f3b5d4c3f2e4fbb66f8234694bb2dfe2"
server_key_iv: "d37894cd3c5ba26d74828a5bd61c5b47"
server_key_version: "encryptionkey@0"
I20260927 02:48:17.898229 9301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:17.927366 9301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:17.929827 9301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:17.929992 9301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:17.930233 9301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:17.930553 9301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:17.930595 9301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.930629 9301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:17.930653 9301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:17.936714 9301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38677
I20260927 02:48:17.936759 9431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38677 every 8 connection(s)
I20260927 02:48:17.937071 9301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:17.937476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9301
I20260927 02:48:17.937741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41915
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38509
--webserver_port=44043
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:48:17.944974 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:17.937697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56223 (local address 127.0.98.62:38509)
0927 02:48:17.937756 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.937759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.938105 (+ 346us) server_negotiation.cc:407] Connection header received
0927 02:48:17.938695 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.938696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.938707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.938750 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.939952 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.940425 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.941828 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.942005 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.942055 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.944921 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56223: BlockingRecv error: recv got EOF from 127.0.98.2:56223 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:48:17.945021 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.937704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:56223)
0927 02:48:17.937968 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.938015 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.938605 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.938849 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.938861 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.939585 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.939870 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.939876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.940510 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.940511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.941751 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.941754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.942077 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.942079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.942223 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.942686 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.942696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.944834 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
W20260927 02:48:17.945212 9432 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38509: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:17.949558 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.945424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:56549)
0927 02:48:17.945472 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.945481 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.945516 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.945628 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.945629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.945761 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.945811 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.945812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.946382 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.946383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.947177 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.947180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.947494 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.947495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.947561 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.947895 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.947900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.949490 (+ 1590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 02:48:17.949615 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:17.945464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56549 (local address 127.0.98.62:38509)
0927 02:48:17.945500 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.945502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.945510 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:48:17.945572 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.945572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.945579 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.945622 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.945885 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.946318 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.947258 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.947434 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.947477 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.949572 (+ 2095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56549: BlockingRecv error: recv got EOF from 127.0.98.2:56549 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:17 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:17.953773 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:17.949910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:51889)
0927 02:48:17.949949 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:17.949959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:17.949998 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:17.950104 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:17.950105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:17.950234 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:17.950264 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.950265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.950832 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.950833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.951627 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:17.951629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.951908 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.951908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:17.951957 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.952267 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:17.952272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:17.953710 (+ 1438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:48:17.953831 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:17.949956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51889 (local address 127.0.98.62:38509)
0927 02:48:17.949989 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:17.949991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:17.949999 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:48:17.950052 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:17.950052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:17.950060 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:17.950100 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:17.950316 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.950756 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.951690 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:17.951853 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:17.951899 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:17.953782 (+ 1883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51889: BlockingRecv error: recv got EOF from 127.0.98.2:51889 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 02:48:17.954071 9432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:48:18.020040 9435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:18.020252 9435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:18.020283 9435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:18.021440 9435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:18.021481 9435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:18.021516 9435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:18.022979 9435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44043
--tserver_master_addrs=127.0.98.62:38509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:18.023209 9435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:18.023427 9435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:18.028131 9443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:18.028231 9435 server_base.cc:1034] running on GCE node
W20260927 02:48:18.028122 9440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:18.028122 9441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:18.028650 9435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:18.028856 9435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:18.029997 9435 hybrid_clock.cc:648] HybridClock initialized: now 1790477298029984 us; error 36 us; skew 500 ppm
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477298, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:18.032399 9435 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 02:48:18.032783 9435 webserver.cc:460] Webserver started at http://127.0.98.3:44043/ using document root <none> and password file <none>
I20260927 02:48:18.032968 9435 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:18.033032 9435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:18.034286 9435 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:18.034824 9450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:18.034964 9435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:18.035051 9435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "d0682b16b06648248ea44e70a70ed4a9"
format_stamp: "Formatted at 2026-09-27 02:48:17 on dist-test-slave-f4rg"
server_key: "ad3c3385c4b92a208abb62cbf08bdeee"
server_key_iv: "2172620dd31968a268c26258c1a66273"
server_key_version: "encryptionkey@0"
I20260927 02:48:18.035367 9435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:18.050870 9435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:18.053012 9435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:18.053158 9435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:18.053376 9435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:18.053663 9435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:18.053695 9435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:18.053723 9435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:18.053747 9435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:18.059324 9435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41915
I20260927 02:48:18.059392 9563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41915 every 8 connection(s)
I20260927 02:48:18.059722 9435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:18.062589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9435
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 02:48:18.067509 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:18.060386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59673 (local address 127.0.98.62:38509)
0927 02:48:18.060452 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:18.060454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:18.060689 (+ 235us) server_negotiation.cc:407] Connection header received
0927 02:48:18.061167 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:18.061168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:18.061178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:18.061230 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:18.062147 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.062629 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.064172 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.064373 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.064426 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.067449 (+ 3023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59673: BlockingRecv error: recv got EOF from 127.0.98.3:59673 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:48:18.067615 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:18.060392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:59673)
0927 02:48:18.060545 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:18.060592 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:18.061084 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:18.061285 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:18.061289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:18.061841 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:18.062050 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.062056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.062732 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.062733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.064102 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.064105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.064443 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.064444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.064570 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.065063 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:18.065073 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:18.067378 (+ 2305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260927 02:48:18.067924 9564 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38509: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 02:48:18.069913 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:18.063116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.0.98.62:38509)
0927 02:48:18.063160 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:18.063162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:18.063207 (+ 45us) server_negotiation.cc:407] Connection header received
0927 02:48:18.063284 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:18.063285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:18.063294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:18.063350 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:18.064084 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.064570 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.065524 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.065710 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.065763 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.066951 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:18.066955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:18.066956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:18.066967 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:18.068676 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:18.068998 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:18.069002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:18.069002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:18.069097 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:18.069261 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:18.069262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:18.069263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:18.069493 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:18.069612 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:18.069715 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:48:18.069817 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:18.071914 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:18.068140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:51643)
0927 02:48:18.068182 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:18.068191 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:18.068223 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:18.068345 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:18.068346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:18.068461 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:18.068501 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.068503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.069066 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.069067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.069559 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.069561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.069909 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.069910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.069975 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.070322 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:18.070327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:18.071851 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 02:48:18.071979 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:18.068191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51643 (local address 127.0.98.62:38509)
0927 02:48:18.068230 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:18.068232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:18.068253 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:48:18.068271 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:18.068271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:18.068280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:18.068341 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:18.068561 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.069004 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.069623 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.069814 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.069862 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.071931 (+ 2069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51643: BlockingRecv error: recv got EOF from 127.0.98.3:51643 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:18.075781 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:18.072153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:59531)
0927 02:48:18.072190 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:18.072199 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:18.072233 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:18.072326 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:18.072327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:18.072438 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:18.072465 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.072466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.073028 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.073029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.073499 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.073500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.073802 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.073802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.073849 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.074147 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:18.074151 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:18.075734 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:48:18.075888 9300 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:18.072199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59531 (local address 127.0.98.62:38509)
0927 02:48:18.072237 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:18.072239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:18.072260 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:48:18.072277 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:18.072278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:18.072286 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:18.072325 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:18.072528 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.072960 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.073577 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.073749 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.073800 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.075831 (+ 2031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59531: BlockingRecv error: recv got EOF from 127.0.98.3:59531 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 02:48:18.075935 9564 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:18.822230 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:18.818146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:43589)
0927 02:48:18.818268 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:18.818278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:18.818326 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:18.818430 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:18.818430 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:18.818575 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:18.818620 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.818622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.819225 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.819226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.819740 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.819741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.820080 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.820081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.820132 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.820533 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:18.820537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:18.822173 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
W20260927 02:48:18.822321 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:18.818218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43589 (local address 127.0.98.62:38509)
0927 02:48:18.818331 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:18.818333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:18.818342 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:18.818362 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:18.818363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:18.818372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:18.818425 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:18.818670 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.819176 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.819825 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.820023 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.820080 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.822256 (+ 2176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43589: BlockingRecv error: recv got EOF from 127.0.98.1:43589 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:18 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:18.958559 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:18.954501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:46805)
0927 02:48:18.954626 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:18.954637 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:18.954702 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:18.954851 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:18.954852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:18.955053 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:18.955109 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.955111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.955709 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.955710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.956239 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:18.956240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.956560 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.956560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:18.956596 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.956986 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:18.956990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:18.958455 (+ 1465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
W20260927 02:48:18.958590 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:18.954561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46805 (local address 127.0.98.62:38509)
0927 02:48:18.954605 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:18.954607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:18.954706 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:48:18.954759 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:18.954760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:18.954773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:18.954839 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:18.955171 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.955652 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.956312 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:18.956505 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:18.956555 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:18.958545 (+ 1990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46805: BlockingRecv error: recv got EOF from 127.0.98.2:46805 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 02:48:19 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:19 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:19.080318 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:19.076241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:43679)
0927 02:48:19.076360 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:19.076369 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:19.076426 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:19.076541 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:19.076542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:19.076685 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:19.076731 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:19.076732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.077318 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.077318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:19.077858 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:19.077860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.078161 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.078161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:19.078196 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:19.078583 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:19.078587 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:19.080251 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
W20260927 02:48:19.080401 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:19.076290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43679 (local address 127.0.98.62:38509)
0927 02:48:19.076379 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:19.076382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:19.076426 (+ 44us) server_negotiation.cc:407] Connection header received
0927 02:48:19.076478 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:19.076479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:19.076491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:19.076528 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:19.076785 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.077265 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.077925 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.078111 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.078158 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:19.080340 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43679: BlockingRecv error: recv got EOF from 127.0.98.3:43679 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 02:48:19 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:19 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:19.827091 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:19.822724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:39819)
0927 02:48:19.822862 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:19.822873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:19.822938 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:19.823108 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:19.823109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:19.823303 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:19.823362 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:19.823364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.823972 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.823973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:19.824507 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:19.824509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.824832 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.824834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:19.824886 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:19.825341 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:19.825345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:19.827034 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260927 02:48:19.827231 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:19.822825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39819 (local address 127.0.98.62:38509)
0927 02:48:19.822934 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:19.822935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:19.822944 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:19.823017 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:19.823017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:19.823029 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:19.823093 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:19.823422 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.823916 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.824567 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.824751 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.824796 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:19.827152 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39819: BlockingRecv error: recv got EOF from 127.0.98.1:39819 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
Sep 27 02:48:19 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:19 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:19.963549 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:19.959028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:41471)
0927 02:48:19.959136 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:19.959145 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:19.959197 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:19.959356 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:19.959357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:19.959514 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:19.959559 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:19.959561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.960260 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.960261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:19.960791 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:19.960793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.961181 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.961183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:19.961237 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:19.961643 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:19.961647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:19.963474 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
W20260927 02:48:19.963598 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:19.959089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41471 (local address 127.0.98.62:38509)
0927 02:48:19.959140 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:19.959142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:19.959183 (+ 41us) server_negotiation.cc:407] Connection header received
0927 02:48:19.959238 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:19.959239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:19.959249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:19.959310 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:19.959638 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.960163 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.960885 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:19.961090 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:19.961149 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:19.963550 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41471: BlockingRecv error: recv got EOF from 127.0.98.2:41471 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 02:48:20 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:20 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:20.084923 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:20.080747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:49755)
0927 02:48:20.080887 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:20.080898 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:20.080957 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:20.081105 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:20.081105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:20.081254 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:20.081300 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:20.081302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.081898 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.081898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:20.082432 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:20.082433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.082759 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.082760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:20.082796 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:20.083238 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:20.083242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:20.084863 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
W20260927 02:48:20.085019 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:20.080860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49755 (local address 127.0.98.62:38509)
0927 02:48:20.080903 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:20.080905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:20.080943 (+ 38us) server_negotiation.cc:407] Connection header received
0927 02:48:20.081033 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:20.081034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:20.081046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:20.081090 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:20.081355 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.081842 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.082506 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.082702 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.082755 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:20.084951 (+ 2196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49755: BlockingRecv error: recv got EOF from 127.0.98.3:49755 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 02:48:20 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:20 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:20.831642 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:20.827504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:43319)
0927 02:48:20.827637 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:20.827649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:20.827705 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:20.827844 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:20.827845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:20.828009 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:20.828055 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:20.828056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.828676 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.828676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:20.829192 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:20.829193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.829473 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.829473 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:20.829508 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:20.829937 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:20.829943 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:20.831566 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260927 02:48:20.831723 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:20.827560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43319 (local address 127.0.98.62:38509)
0927 02:48:20.827685 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:20.827687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:20.827696 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:20.827761 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:20.827761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:20.827773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:20.827832 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:20.828111 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.828619 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.829251 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.829429 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.829481 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:20.831665 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43319: BlockingRecv error: recv got EOF from 127.0.98.1:43319 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
Sep 27 02:48:20 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:20 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:20.968307 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:20.963950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:48317)
0927 02:48:20.964075 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:20.964088 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:20.964155 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:20.964270 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:20.964271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:20.964426 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:20.964474 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:20.964476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.965072 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.965072 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:20.965588 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:20.965590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.965915 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.965917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:20.965976 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:20.966463 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:20.966467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:20.968212 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
W20260927 02:48:20.968372 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:20.964057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48317 (local address 127.0.98.62:38509)
0927 02:48:20.964114 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:20.964115 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:20.964139 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:48:20.964199 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:20.964200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:20.964210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:20.964254 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:20.964522 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.965016 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.965640 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:20.965842 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:20.965901 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:20.968321 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48317: BlockingRecv error: recv got EOF from 127.0.98.2:48317 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:48:21 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:21 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:21.089655 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:21.085388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:40141)
0927 02:48:21.085543 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:21.085554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:21.085611 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:21.085712 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:21.085713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:21.085885 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:21.085938 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:21.085940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.086610 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.086611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:21.087236 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:21.087238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.087532 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.087533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:21.087568 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:21.087950 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:21.087958 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:21.089590 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260927 02:48:21.089762 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:21.085448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40141 (local address 127.0.98.62:38509)
0927 02:48:21.085495 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:21.085498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:21.085597 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:48:21.085650 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:21.085651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:21.085662 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:21.085702 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:21.086012 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.086546 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.087291 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.087481 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.087531 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:21.089699 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40141: BlockingRecv error: recv got EOF from 127.0.98.3:40141 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 02:48:21 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:21 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:21.836267 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:21.832077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:53963)
0927 02:48:21.832186 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:21.832196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:21.832246 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:21.832401 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:21.832402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:21.832554 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:21.832601 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:21.832602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.833250 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.833251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:21.833772 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:21.833773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.834091 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.834092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:21.834126 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:21.834584 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:21.834589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:21.836209 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
W20260927 02:48:21.836366 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:21.832169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53963 (local address 127.0.98.62:38509)
0927 02:48:21.832297 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:21.832298 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:21.832306 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:48:21.832325 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:21.832325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:21.832334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:21.832388 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:21.832663 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.833189 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.833842 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.834038 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.834089 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:21.836301 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53963: BlockingRecv error: recv got EOF from 127.0.98.1:53963 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
Sep 27 02:48:21 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:21 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:21.972885 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:21.968762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:42719)
0927 02:48:21.968912 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:21.968923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:21.968992 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:21.969102 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:21.969102 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:21.969250 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:21.969296 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:21.969298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.969886 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.969887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:21.970430 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:21.970431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.970714 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.970715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:21.970754 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:21.971190 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:21.971194 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:21.972826 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
W20260927 02:48:21.972973 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:21.968822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42719 (local address 127.0.98.62:38509)
0927 02:48:21.968880 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:21.968883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:21.968983 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:48:21.969031 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:21.969031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:21.969041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:21.969088 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:21.969343 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.969832 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.970485 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:21.970658 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:21.970701 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:21.972911 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42719: BlockingRecv error: recv got EOF from 127.0.98.2:42719 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 02:48:22 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:22 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:22.094231 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:22.090153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:57915)
0927 02:48:22.090271 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:22.090281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:22.090337 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:22.090489 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:22.090490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:22.090638 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:22.090683 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:22.090685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.091308 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.091309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:22.091835 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:22.091836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.092138 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.092138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:22.092172 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:22.092620 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:22.092624 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:22.094171 (+ 1547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260927 02:48:22.094313 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:22.090224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57915 (local address 127.0.98.62:38509)
0927 02:48:22.090295 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:22.090297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:22.090325 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:48:22.090400 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:22.090400 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:22.090413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:22.090452 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:22.090741 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.091243 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.091905 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.092085 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.092136 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:22.094254 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57915: BlockingRecv error: recv got EOF from 127.0.98.3:57915 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 02:48:22 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:22 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:22.846110 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:22.836738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:52557)
0927 02:48:22.836847 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:22.836859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:22.836925 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:22.837036 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:22.837037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:22.837181 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:22.837228 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:22.837229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.837830 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.837831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:22.838365 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:22.838366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.838672 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.838673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:22.838707 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:22.844281 (+ 5574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:22.844286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:22.846015 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
W20260927 02:48:22.846190 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:22.836800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52557 (local address 127.0.98.62:38509)
0927 02:48:22.836927 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:22.836928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:22.836955 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:48:22.836975 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:22.836976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:22.836985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:22.837030 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:22.837279 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.837771 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.838429 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.838620 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.838669 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:22.846122 (+ 7453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52557: BlockingRecv error: recv got EOF from 127.0.98.1:52557 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
Sep 27 02:48:22 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:22 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:22.977524 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:22.973361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:56609)
0927 02:48:22.973499 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:22.973512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:22.973570 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:22.973677 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:22.973677 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:22.973828 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:22.973883 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:22.973885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.974541 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.974542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:22.975101 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:22.975102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.975441 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.975442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:22.975498 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:22.975896 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:22.975901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:22.977453 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260927 02:48:22.977599 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:22.973493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56609 (local address 127.0.98.62:38509)
0927 02:48:22.973550 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:22.973552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:22.973561 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:22.973616 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:22.973617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:22.973627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:22.973671 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:22.973935 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.974475 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.975178 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:22.975371 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:22.975420 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:22.977540 (+ 2120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56609: BlockingRecv error: recv got EOF from 127.0.98.2:56609 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 02:48:23 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:23 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:23.098793 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:23.094662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:53121)
0927 02:48:23.094795 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:23.094829 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:23.094888 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:23.095023 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:23.095023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:23.095166 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:23.095213 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:23.095215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.095814 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.095815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:23.096357 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:23.096359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.096643 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.096643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:23.096678 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:23.097083 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:23.097087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:23.098700 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260927 02:48:23.098886 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:23.094780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53121 (local address 127.0.98.62:38509)
0927 02:48:23.094840 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:23.094842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:23.094876 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:48:23.094942 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:23.094943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:23.094954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:23.095010 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:23.095263 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.095748 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.096415 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.096593 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.096639 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:23.098796 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53121: BlockingRecv error: recv got EOF from 127.0.98.3:53121 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 02:48:23 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:23 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:23.850685 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:23.846586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:32831)
0927 02:48:23.846725 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:23.846738 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:23.846836 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:23.846977 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:23.846978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:23.847128 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:23.847175 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:23.847176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.847775 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.847775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:23.848310 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:23.848312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.848658 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.848658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:23.848697 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:23.849101 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:23.849105 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:23.850615 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
W20260927 02:48:23.850768 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:23.846666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32831 (local address 127.0.98.62:38509)
0927 02:48:23.846794 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:23.846796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:23.846824 (+ 28us) server_negotiation.cc:407] Connection header received
0927 02:48:23.846898 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:23.846899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:23.846916 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:23.846961 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:23.847224 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.847713 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.848422 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.848606 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.848651 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:23.850703 (+ 2052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32831: BlockingRecv error: recv got EOF from 127.0.98.1:32831 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
Sep 27 02:48:23 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:23 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:23.982100 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:23.977996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:40555)
0927 02:48:23.978121 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:23.978130 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:23.978189 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:23.978312 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:23.978313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:23.978469 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:23.978516 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:23.978518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.979152 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.979152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:23.979673 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:23.979674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.979965 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.979966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:23.980001 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:23.980431 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:23.980435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:23.982041 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
W20260927 02:48:23.982193 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:23.978069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40555 (local address 127.0.98.62:38509)
0927 02:48:23.978128 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:23.978129 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:23.978182 (+ 53us) server_negotiation.cc:407] Connection header received
0927 02:48:23.978235 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:23.978236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:23.978246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:23.978301 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:23.978577 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.979085 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.979726 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:23.979914 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:23.979963 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:23.982130 (+ 2167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40555: BlockingRecv error: recv got EOF from 127.0.98.2:40555 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 02:48:24 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:24 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:24.103559 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:24.099278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:34867)
0927 02:48:24.099404 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:24.099414 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:24.099471 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:24.099568 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:24.099568 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:24.099714 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:24.099760 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:24.099762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.100347 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.100348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:24.100933 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:24.100936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.101238 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.101239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:24.101274 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:24.101745 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:24.101750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:24.103481 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
W20260927 02:48:24.103632 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:24.099323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34867 (local address 127.0.98.62:38509)
0927 02:48:24.099377 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:24.099379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:24.099456 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:48:24.099507 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:24.099507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:24.099520 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:24.099561 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:24.099807 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.100285 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.100999 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.101185 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.101236 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:24.103577 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34867: BlockingRecv error: recv got EOF from 127.0.98.3:34867 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 02:48:24 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:24 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:24.855785 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:24.851149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:34101)
0927 02:48:24.851316 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:24.851326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:24.851394 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:24.851518 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:24.851519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:24.851675 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:24.851725 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:24.851726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.852332 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.852332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:24.852851 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:24.852852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.853284 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.853285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:24.853330 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:24.853813 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:24.853819 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:24.855721 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
W20260927 02:48:24.855872 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:24.851219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34101 (local address 127.0.98.62:38509)
0927 02:48:24.851316 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:24.851318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:24.851388 (+ 70us) server_negotiation.cc:407] Connection header received
0927 02:48:24.851437 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:24.851438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:24.851449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:24.851506 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:24.851774 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.852277 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.852941 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.853149 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.853255 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:24.855805 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34101: BlockingRecv error: recv got EOF from 127.0.98.1:34101 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
Sep 27 02:48:24 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:24 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:24.987246 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:24.982611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:40795)
0927 02:48:24.982762 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:24.982773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:24.982862 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:24.982999 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:24.982999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:24.983211 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:24.983271 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:24.983274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.984044 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.984045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:24.984576 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:24.984577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.984934 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.984935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:24.984975 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:24.985457 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:24.985462 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:24.987187 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
W20260927 02:48:24.987370 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:24.982753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40795 (local address 127.0.98.62:38509)
0927 02:48:24.982843 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:24.982845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:24.982855 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:48:24.982926 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:24.982927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:24.982940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:24.982983 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:24.983377 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.983975 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.984669 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:24.984870 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:24.984931 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:24.987299 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40795: BlockingRecv error: recv got EOF from 127.0.98.2:40795 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 02:48:25 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:25 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:25.108222 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:25.104035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:44293)
0927 02:48:25.104171 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:25.104182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:25.104249 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:25.104373 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:25.104374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:25.104553 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:25.104600 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:25.104602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.105238 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.105239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:25.105762 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:25.105763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.106075 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.106076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:25.106128 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:25.106535 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:25.106538 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:25.108134 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
W20260927 02:48:25.108292 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:25.104101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44293 (local address 127.0.98.62:38509)
0927 02:48:25.104174 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:25.104177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:25.104237 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:48:25.104308 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:25.104309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:25.104322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:25.104360 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:25.104666 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.105170 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.105828 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.106019 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.106070 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:25.108239 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44293: BlockingRecv error: recv got EOF from 127.0.98.3:44293 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 02:48:25 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:25 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:25.860497 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:25.856236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:52905)
0927 02:48:25.856359 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:25.856372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:25.856427 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:25.856604 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:25.856604 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:25.856749 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:25.856795 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:25.856797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.857421 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.857421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:25.857963 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:25.857964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.858275 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.858276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:25.858351 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:25.858746 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:25.858750 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:25.860439 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
W20260927 02:48:25.860620 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:25.856401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52905 (local address 127.0.98.62:38509)
0927 02:48:25.856503 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:25.856504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:25.856511 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:48:25.856531 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:25.856532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:25.856541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:25.856595 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:25.856853 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.857338 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.858021 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.858203 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.858246 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:25.860539 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52905: BlockingRecv error: recv got EOF from 127.0.98.1:52905 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":42,"threads_started":1}
Sep 27 02:48:25 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:25 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:25.992228 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:25.987771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:38207)
0927 02:48:25.987900 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:25.987910 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:25.987968 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:25.988090 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:25.988091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:25.988258 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:25.988333 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:25.988335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.989094 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.989095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:25.989637 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:25.989639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.989929 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.989929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:25.989965 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:25.990363 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:25.990367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:25.992140 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260927 02:48:25.992296 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:25.987822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38207 (local address 127.0.98.62:38509)
0927 02:48:25.987873 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:25.987875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:25.987964 (+ 89us) server_negotiation.cc:407] Connection header received
0927 02:48:25.988028 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:25.988029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:25.988039 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:25.988078 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:25.988396 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.989020 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.989699 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:25.989879 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:25.989925 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:25.992236 (+ 2311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38207: BlockingRecv error: recv got EOF from 127.0.98.2:38207 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 02:48:26 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:26 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:26.113034 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:26.108728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:49245)
0927 02:48:26.108858 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:26.108867 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:26.108934 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:26.109044 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:26.109044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:26.109244 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:26.109291 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:26.109293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.109902 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.109903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:26.110588 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:26.110590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.111007 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.111008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:26.111066 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:26.111441 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:26.111444 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:26.112976 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
W20260927 02:48:26.113119 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:26.108841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49245 (local address 127.0.98.62:38509)
0927 02:48:26.108885 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:26.108887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:26.108919 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:48:26.108976 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:26.108977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:26.108987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:26.109032 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:26.109348 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.109839 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.110641 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.110933 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.110984 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:26.113060 (+ 2076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49245: BlockingRecv error: recv got EOF from 127.0.98.3:49245 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 02:48:26 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:26 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:26.865319 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:26.860915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:35129)
0927 02:48:26.861036 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:26.861049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:26.861103 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:26.861244 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:26.861245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:26.861404 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:26.861453 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:26.861455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.862083 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.862084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:26.862617 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:26.862620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.863084 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.863085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:26.863148 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:26.863571 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:26.863574 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:26.865242 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
W20260927 02:48:26.865394 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:26.860959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35129 (local address 127.0.98.62:38509)
0927 02:48:26.861105 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:26.861106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:26.861142 (+ 36us) server_negotiation.cc:407] Connection header received
0927 02:48:26.861167 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:26.861167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:26.861178 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:26.861231 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:26.861513 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.862024 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.862693 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.863007 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.863065 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:26.865333 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35129: BlockingRecv error: recv got EOF from 127.0.98.1:35129 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
Sep 27 02:48:26 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:26 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:26.996883 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:26.992674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:54091)
0927 02:48:26.992813 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:26.992824 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:26.992899 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:26.993018 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:26.993019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:26.993179 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:26.993223 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:26.993226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.993863 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.993865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:26.994373 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:26.994374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.994687 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.994688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:26.994740 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:26.995181 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:26.995185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:26.996805 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
W20260927 02:48:26.996955 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:26.992787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54091 (local address 127.0.98.62:38509)
0927 02:48:26.992866 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:26.992868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:26.992881 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:48:26.992952 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:26.992953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:26.992966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:26.993013 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:26.993287 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.993787 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.994440 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:26.994629 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:26.994683 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:26.996903 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54091: BlockingRecv error: recv got EOF from 127.0.98.2:54091 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 02:48:27 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:27 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:27.117722 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:27.113476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:44061)
0927 02:48:27.113596 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:27.113608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:27.113677 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:27.113799 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:27.113800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:27.113966 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:27.114011 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:27.114012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.114639 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.114640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:27.115178 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:27.115179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.115500 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.115501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:27.115553 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:27.115989 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:27.115993 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:27.117664 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260927 02:48:27.117842 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:27.113569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44061 (local address 127.0.98.62:38509)
0927 02:48:27.113623 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:27.113625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:27.113660 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:48:27.113725 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:27.113726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:27.113740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:27.113788 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:27.114080 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.114568 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.115252 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.115443 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.115495 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:27.117749 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44061: BlockingRecv error: recv got EOF from 127.0.98.3:44061 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 02:48:27 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:27 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:27.878469 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:27.865856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:46905)
0927 02:48:27.865997 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:27.866007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:27.866063 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:27.866181 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:27.866181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:27.866330 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:27.866378 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:27.866379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.867060 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.867061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:27.867641 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:27.867644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.868053 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.868054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:27.868124 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:27.876665 (+ 8541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:27.876670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:27.878408 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
W20260927 02:48:27.878566 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:27.865969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46905 (local address 127.0.98.62:38509)
0927 02:48:27.866071 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:27.866073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:27.866081 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:48:27.866101 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:27.866101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:27.866111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:27.866175 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:27.866436 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.866989 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.867734 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.867968 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.868026 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:27.878502 (+ 10476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46905: BlockingRecv error: recv got EOF from 127.0.98.1:46905 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":43,"threads_started":1}
Sep 27 02:48:27 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:27 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:28.002056 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:27.997408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:33433)
0927 02:48:27.997562 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:27.997574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:27.997641 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:27.997773 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:27.997774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:27.997928 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:27.997973 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:27.997974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.998670 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.998671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:27.999257 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:27.999260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.999638 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.999639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:27.999697 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:28.000216 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:28.000222 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:28.001997 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":69,"threads_started":1}
W20260927 02:48:28.002195 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:27.997509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33433 (local address 127.0.98.62:38509)
0927 02:48:27.997569 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:27.997571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:27.997628 (+ 57us) server_negotiation.cc:407] Connection header received
0927 02:48:27.997699 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:27.997700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:27.997711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:27.997750 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:27.998043 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.998584 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.999347 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:27.999539 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:27.999611 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:28.002085 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33433: BlockingRecv error: recv got EOF from 127.0.98.2:33433 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 02:48:28 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:28 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:28.122761 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:28.118213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:41085)
0927 02:48:28.118390 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:28.118403 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:28.118466 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:28.118645 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:28.118646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:28.118844 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:28.118901 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:28.118903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.119586 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.119587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:28.120105 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:28.120106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.120437 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.120438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:28.120492 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:28.120902 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:28.120906 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:28.122703 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
W20260927 02:48:28.122913 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:28.118374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41085 (local address 127.0.98.62:38509)
0927 02:48:28.118451 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:28.118453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:28.118462 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:28.118530 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:28.118531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:28.118543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:28.118582 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:28.118988 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.119508 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.120180 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.120362 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.120422 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:28.122845 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41085: BlockingRecv error: recv got EOF from 127.0.98.3:41085 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 02:48:28 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:28 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:28.883284 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:28.878939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:57761)
0927 02:48:28.879080 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:28.879091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:28.879157 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:28.879286 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:28.879286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:28.879441 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:28.879489 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:28.879490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.880085 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.880086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:28.880643 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:28.880645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.880963 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.880965 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:28.881024 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:28.881471 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:28.881475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:28.883202 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
W20260927 02:48:28.883391 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:28.879020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57761 (local address 127.0.98.62:38509)
0927 02:48:28.879124 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:28.879126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:28.879136 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:48:28.879210 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:28.879210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:28.879220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:28.879275 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:28.879537 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.880029 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.880705 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:28.880893 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:28.880936 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:28.883329 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57761: BlockingRecv error: recv got EOF from 127.0.98.1:57761 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":45,"threads_started":1}
Sep 27 02:48:29 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:29 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:29.006651 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:29.002460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:59519)
0927 02:48:29.002595 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:29.002606 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:29.002667 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:29.002786 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:29.002787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:29.002957 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:29.003005 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:29.003006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.003604 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.003605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:29.004126 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:29.004128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.004462 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.004462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:29.004503 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:29.004969 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:29.004974 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:29.006593 (+ 1619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
W20260927 02:48:29.006718 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:29.002562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59519 (local address 127.0.98.62:38509)
0927 02:48:29.002619 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:29.002621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:29.002652 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:48:29.002720 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:29.002721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:29.002730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:29.002777 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:29.003053 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.003545 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.004226 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.004413 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.004455 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:29.006666 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59519: BlockingRecv error: recv got EOF from 127.0.98.2:59519 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 02:48:29 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:29 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:29.127404 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:29.123284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:58533)
0927 02:48:29.123429 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:29.123439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:29.123509 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:29.123612 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:29.123613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:29.123764 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:29.123811 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:29.123812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.124403 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.124404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:29.124932 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:29.124934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.125220 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.125221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:29.125266 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:29.125660 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:29.125663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:29.127345 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
W20260927 02:48:29.127482 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:29.123387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58533 (local address 127.0.98.62:38509)
0927 02:48:29.123433 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:29.123435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:29.123492 (+ 57us) server_negotiation.cc:407] Connection header received
0927 02:48:29.123550 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:29.123550 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:29.123561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:29.123605 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:29.123860 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.124336 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.124990 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.125159 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.125200 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:29.127424 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58533: BlockingRecv error: recv got EOF from 127.0.98.3:58533 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 02:48:29 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:29 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:29.888047 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:29.883704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:48073)
0927 02:48:29.883830 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:29.883843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:29.883900 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:29.884051 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:29.884052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:29.884205 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:29.884251 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:29.884253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.884859 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.884860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:29.885393 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:29.885394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.885716 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.885717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:29.885751 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:29.886328 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:29.886332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:29.887991 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
W20260927 02:48:29.888165 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:29.883768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48073 (local address 127.0.98.62:38509)
0927 02:48:29.883925 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:29.883927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:29.883936 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:29.883957 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:29.883958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:29.883968 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:29.884037 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:29.884311 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.884798 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.885466 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:29.885660 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:29.885716 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:29.888093 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48073: BlockingRecv error: recv got EOF from 127.0.98.1:48073 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
Sep 27 02:48:30 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:30 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:30.011504 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:30.007108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:42271)
0927 02:48:30.007243 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:30.007255 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:30.007322 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:30.007470 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:30.007471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:30.007667 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:30.007715 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:30.007717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.008475 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.008476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:30.009014 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:30.009015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.009292 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.009293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:30.009331 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:30.009787 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:30.009793 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:30.011438 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
W20260927 02:48:30.011569 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:30.007202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42271 (local address 127.0.98.62:38509)
0927 02:48:30.007253 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:30.007255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:30.007321 (+ 66us) server_negotiation.cc:407] Connection header received
0927 02:48:30.007372 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:30.007372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:30.007383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:30.007423 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:30.007797 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.008411 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.009072 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.009243 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.009286 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:30.011510 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42271: BlockingRecv error: recv got EOF from 127.0.98.2:42271 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 02:48:30 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:30 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:30.132200 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:30.127905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:36637)
0927 02:48:30.128075 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:30.128086 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:30.128159 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:30.128298 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:30.128299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:30.128469 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:30.128515 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:30.128517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.129143 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.129144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:30.129673 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:30.129675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.129979 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.129980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:30.130033 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:30.130450 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:30.130454 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:30.132113 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
W20260927 02:48:30.132259 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:30.127982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36637 (local address 127.0.98.62:38509)
0927 02:48:30.128051 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:30.128054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:30.128139 (+ 85us) server_negotiation.cc:407] Connection header received
0927 02:48:30.128220 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:30.128222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:30.128235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:30.128289 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:30.128582 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.129068 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.129746 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.129923 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.129975 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:30.132206 (+ 2231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36637: BlockingRecv error: recv got EOF from 127.0.98.3:36637 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 02:48:30 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:30 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:30.892513 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:30.888471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:45073)
0927 02:48:30.888590 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:30.888601 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:30.888651 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:30.888788 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:30.888789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:30.888945 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:30.888988 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:30.888989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.889574 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.889575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:30.890089 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:30.890090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.890394 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.890394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:30.890428 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:30.890840 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:30.890844 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:30.892449 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
W20260927 02:48:30.892596 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:30.888540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45073 (local address 127.0.98.62:38509)
0927 02:48:30.888669 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:30.888671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:30.888680 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:30.888700 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:30.888700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:30.888710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:30.888771 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:30.889041 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.889524 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.890140 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:30.890344 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:30.890389 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:30.892535 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45073: BlockingRecv error: recv got EOF from 127.0.98.1:45073 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
Sep 27 02:48:31 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:31 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:31.016165 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:31.011943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:50125)
0927 02:48:31.012098 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:31.012110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:31.012170 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:31.012312 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:31.012313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:31.012461 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:31.012509 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:31.012511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.013096 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.013096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:31.013629 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:31.013631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.013912 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.013913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:31.013949 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:31.014408 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:31.014413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:31.016105 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
W20260927 02:48:31.016247 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:31.012063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50125 (local address 127.0.98.62:38509)
0927 02:48:31.012125 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:31.012126 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:31.012155 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:48:31.012241 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:31.012242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:31.012251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:31.012299 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:31.012560 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.013044 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.013685 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.013867 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.013908 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:31.016190 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50125: BlockingRecv error: recv got EOF from 127.0.98.2:50125 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 02:48:31 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:31 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:31.136992 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:31.132668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:53087)
0927 02:48:31.132784 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:31.132793 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:31.132852 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:31.133010 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:31.133012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:31.133207 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:31.133261 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:31.133263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.133888 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.133889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:31.134409 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:31.134410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.134714 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.134715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:31.134768 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:31.135210 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:31.135214 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:31.136916 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
W20260927 02:48:31.137061 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:31.132731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53087 (local address 127.0.98.62:38509)
0927 02:48:31.132809 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:31.132811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:31.132840 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:48:31.132908 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:31.132908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:31.132920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:31.132975 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:31.133332 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.133821 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.134474 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.134658 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.134710 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:31.137007 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53087: BlockingRecv error: recv got EOF from 127.0.98.3:53087 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 02:48:31 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:31 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:31.897228 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:31.892972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:51837)
0927 02:48:31.893111 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:31.893124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:31.893186 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:31.893331 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:31.893331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:31.893515 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:31.893562 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:31.893564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.894286 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.894287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:31.894830 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:31.894831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.895147 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.895148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:31.895205 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:31.895585 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:31.895589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:31.897168 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260927 02:48:31.897341 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:31.893053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51837 (local address 127.0.98.62:38509)
0927 02:48:31.893169 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:31.893171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:31.893179 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:48:31.893252 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:31.893252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:31.893262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:31.893306 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:31.893631 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.894215 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.894894 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:31.895091 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:31.895143 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:31.897266 (+ 2123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51837: BlockingRecv error: recv got EOF from 127.0.98.1:51837 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
Sep 27 02:48:32 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:32 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:32.021044 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:32.016648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:37233)
0927 02:48:32.016787 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:32.016797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:32.016853 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:32.016985 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:32.016986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:32.017134 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:32.017180 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:32.017181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.017806 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.017807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:32.018345 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:32.018347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.018674 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.018675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:32.018717 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:32.019213 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:32.019219 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:32.020983 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
W20260927 02:48:32.021171 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:32.016736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37233 (local address 127.0.98.62:38509)
0927 02:48:32.016791 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:32.016793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:32.016840 (+ 47us) server_negotiation.cc:407] Connection header received
0927 02:48:32.016914 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:32.016915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:32.016924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:32.016965 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:32.017263 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.017750 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.018411 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.018608 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.018657 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:32.021109 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37233: BlockingRecv error: recv got EOF from 127.0.98.2:37233 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 02:48:32 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:32 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:32.142097 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:32.137548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.3:59059)
0927 02:48:32.137705 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:32.137718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:32.137781 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:32.137905 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:32.137906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:32.138075 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:32.138125 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:32.138127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.138850 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.138851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:32.139374 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:32.139375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.139714 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.139715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:32.139768 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:32.140208 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:32.140212 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:32.142038 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
W20260927 02:48:32.142256 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:32.137727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59059 (local address 127.0.98.62:38509)
0927 02:48:32.137792 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:32.137794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:32.137804 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:48:32.137829 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:32.137830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:32.137843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:32.137892 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:32.138209 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.138754 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.139452 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.139657 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.139709 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:32.142196 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59059: BlockingRecv error: recv got EOF from 127.0.98.3:59059 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 02:48:32 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:32 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:32.905047 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:32.897710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.1:37411)
0927 02:48:32.897818 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:32.897830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:32.897893 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:32.898015 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:32.898016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:32.898196 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:32.898246 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:32.898248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.898874 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.898874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:32.899424 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:32.899426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.899750 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.899751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:32.899795 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:32.903298 (+ 3503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:32.903304 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:32.904975 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
W20260927 02:48:32.905135 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:32.897785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37411 (local address 127.0.98.62:38509)
0927 02:48:32.897905 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:32.897907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:32.897914 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:48:32.897935 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:32.897936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:32.897946 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:32.898000 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:32.898301 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.898785 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.899489 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:32.899685 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:32.899741 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:32.905065 (+ 5324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37411: BlockingRecv error: recv got EOF from 127.0.98.1:37411 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
Sep 27 02:48:33 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 02:48:33 dist-test-slave-f4rg krb5kdc[8566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 02:48:33.025816 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.021536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38509 (local address 127.0.98.2:34303)
0927 02:48:33.021679 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:33.021690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:33.021745 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:33.021859 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:33.021859 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:33.022011 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:33.022061 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.022063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.022669 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.022670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.023253 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.023256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.023571 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.023572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.023623 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.024106 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:33.024110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:33.025747 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38509: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":53,"threads_started":1}
W20260927 02:48:33.025898 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:33.021668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34303 (local address 127.0.98.62:38509)
0927 02:48:33.021723 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.021725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.021734 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:33.021790 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.021791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.021800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.021845 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.022117 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.022605 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.023316 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.023493 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.023537 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.025841 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34303: BlockingRecv error: recv got EOF from 127.0.98.2:34303 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:48:33.072160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9149
I20260927 02:48:33.075958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9301
I20260927 02:48:33.079594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9435
I20260927 02:48:33.082979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9079
2026-09-27T02:48:33Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17274 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T02:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:33Z Disabled control of system clock
I20260927 02:48:33.108080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40309
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 02:48:33.185763 9630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:33.185976 9630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:33.185994 9630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:33.187168 9630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:33.187208 9630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:33.187220 9630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:33.187230 9630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:33.187242 9630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:33.188575 9630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:33.188776 9630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:33.189013 9630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:33.193763 9635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.193921 9636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.193817 9638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.193977 9630 server_base.cc:1034] running on GCE node
I20260927 02:48:33.194205 9630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:33.194486 9630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:33.195619 9630 hybrid_clock.cc:648] HybridClock initialized: now 1790477313195606 us; error 30 us; skew 500 ppm
I20260927 02:48:33.196700 9630 webserver.cc:460] Webserver started at http://127.0.98.62:38119/ using document root <none> and password file <none>
I20260927 02:48:33.196909 9630 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:33.196955 9630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:33.197075 9630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:33.198071 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "31b407de5b9646bf99895933c62c8f33"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "95dc54734b8fafa68a2f4cf38c7f35d8"
server_key_iv: "9dc8d86866f9a1ada841b54a8e338867"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.198396 9630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "31b407de5b9646bf99895933c62c8f33"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "95dc54734b8fafa68a2f4cf38c7f35d8"
server_key_iv: "9dc8d86866f9a1ada841b54a8e338867"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.199608 9630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:48:33.200241 9644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.200384 9630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:33.200462 9630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "31b407de5b9646bf99895933c62c8f33"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "95dc54734b8fafa68a2f4cf38c7f35d8"
server_key_iv: "9dc8d86866f9a1ada841b54a8e338867"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.200520 9630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:33.215039 9630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:33.215358 9630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:33.215490 9630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:33.219255 9630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40309
I20260927 02:48:33.219290 9696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40309 every 8 connection(s)
I20260927 02:48:33.219604 9630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:33.220261 9697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:33.221824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9630
I20260927 02:48:33.221911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:33.222084 392 external_mini_cluster.cc:1428] Setting key bff67e5961a5858ca00566d9a6551ff2
I20260927 02:48:33.223026 9697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33: Bootstrap starting.
I20260927 02:48:33.223639 9697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:33.223923 9697 log.cc:824] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:33.224527 9697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33: No bootstrap required, opened a new log
I20260927 02:48:33.226255 9697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b407de5b9646bf99895933c62c8f33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40309 } }
I20260927 02:48:33.226505 9697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:33.226567 9697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b407de5b9646bf99895933c62c8f33, State: Initialized, Role: FOLLOWER
I20260927 02:48:33.226697 9697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b407de5b9646bf99895933c62c8f33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40309 } }
I20260927 02:48:33.226787 9697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:33.226843 9697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:33.226893 9697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:33.227394 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.222605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.0.98.62:40309)
0927 02:48:33.222770 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.222778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.222802 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:48:33.223291 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.223298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.223360 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.223476 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.223708 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.224622 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.225363 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.225767 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.225888 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.226546 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.226550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.226551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.226743 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.226776 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.226787 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.226907 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.227107 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.227185 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260927 02:48:33.227588 9697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b407de5b9646bf99895933c62c8f33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40309 } }
I20260927 02:48:33.227684 9697 leader_election.cc:302] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31b407de5b9646bf99895933c62c8f33; no voters:
I20260927 02:48:33.227829 9697 leader_election.cc:288] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:33.227869 9702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:33.228082 9697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:33.228152 9702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [term 1 LEADER]: Becoming Leader. State: Replica: 31b407de5b9646bf99895933c62c8f33, State: Running, Role: LEADER
I20260927 02:48:33.228240 9702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b407de5b9646bf99895933c62c8f33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40309 } }
I20260927 02:48:33.228549 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31b407de5b9646bf99895933c62c8f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b407de5b9646bf99895933c62c8f33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40309 } } }
I20260927 02:48:33.228570 9704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31b407de5b9646bf99895933c62c8f33. Latest consensus state: current_term: 1 leader_uuid: "31b407de5b9646bf99895933c62c8f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b407de5b9646bf99895933c62c8f33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40309 } } }
I20260927 02:48:33.228653 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:33.228659 9704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:33.229063 9711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:33.229460 9711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:33.230762 9711 catalog_manager.cc:1347] Generated new cluster ID: 9701c81532bb41b8b412b694bca4260f
I20260927 02:48:33.230800 9711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:33.264250 9711 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:33.264776 9711 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:33.270339 9711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31b407de5b9646bf99895933c62c8f33: Generated new TSK 0
I20260927 02:48:33.270516 9711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:33.281807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40309
--builtin_ntp_servers=127.0.98.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 02:48:33.380204 9721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:33.380401 9721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:33.380420 9721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:33.381491 9721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:33.381524 9721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:33.381546 9721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:33.382783 9721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:33.382973 9721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:33.383211 9721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:33.387830 9727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.387840 9729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.387890 9721 server_base.cc:1034] running on GCE node
W20260927 02:48:33.387849 9726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.388260 9721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:33.388453 9721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:33.389613 9721 hybrid_clock.cc:648] HybridClock initialized: now 1790477313389597 us; error 28 us; skew 500 ppm
I20260927 02:48:33.390789 9721 webserver.cc:460] Webserver started at http://127.0.98.1:41437/ using document root <none> and password file <none>
I20260927 02:48:33.391005 9721 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:33.391052 9721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:33.391170 9721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:33.392185 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "efd48a3806344db7b7d32c00cb43c979"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "538a3509e7ddfa9b6a70150c800bed31"
server_key_iv: "3ac3712bd29dc6bfc7ba21166191f8a5"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.392503 9721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "efd48a3806344db7b7d32c00cb43c979"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "538a3509e7ddfa9b6a70150c800bed31"
server_key_iv: "3ac3712bd29dc6bfc7ba21166191f8a5"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.393638 9721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:48:33.394220 9735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.394375 9721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:33.394450 9721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "efd48a3806344db7b7d32c00cb43c979"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "538a3509e7ddfa9b6a70150c800bed31"
server_key_iv: "3ac3712bd29dc6bfc7ba21166191f8a5"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.394510 9721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:33.413391 9721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:33.413702 9721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:33.413828 9721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:33.414033 9721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:33.414338 9721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:33.414371 9721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.414400 9721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:33.414423 9721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.420815 9721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41561
I20260927 02:48:33.420931 9848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41561 every 8 connection(s)
I20260927 02:48:33.421283 9721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:48:33.426928 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.422012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47045 (local address 127.0.98.62:40309)
0927 02:48:33.422074 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.422076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.422369 (+ 293us) server_negotiation.cc:407] Connection header received
0927 02:48:33.423087 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.423089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.423101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.423133 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.424220 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.424904 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.425828 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.426005 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.426047 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.426701 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.426702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.426702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.426706 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.426721 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.426725 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.426764 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.426830 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.426853 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:48:33.427140 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.422015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40309 (local address 127.0.98.1:47045)
0927 02:48:33.422224 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:33.422271 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:33.422961 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:33.423292 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:33.423295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:33.423303 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:33.424131 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.424138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.424973 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.424974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.425758 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.425760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.426049 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.426050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.426147 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.426586 (+ 439us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:33.426587 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:33.426615 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:33.426636 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:33.426764 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:33.426767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:33.426769 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:48:33.426835 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:48:33.426928 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260927 02:48:33.427160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9721
I20260927 02:48:33.427241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:33.427449 392 external_mini_cluster.cc:1428] Setting key 79a01f23cdf7d0b1405a3f26aa21c71b
I20260927 02:48:33.427600 9849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40309
I20260927 02:48:33.427695 9849 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:33.427882 9849 heartbeater.cc:511] Master 127.0.98.62:40309 requested a full tablet report, sending...
I20260927 02:48:33.428342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40309
--builtin_ntp_servers=127.0.98.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:48:33.428364 9661 ts_manager.cc:194] Registered new tserver with Master: efd48a3806344db7b7d32c00cb43c979 (127.0.98.1:41561)
I20260927 02:48:33.428964 9661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47045
W20260927 02:48:33.506508 9852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:33.506722 9852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:33.506742 9852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:33.507890 9852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:33.507928 9852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:33.507952 9852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:33.509207 9852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:33.509392 9852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:33.509642 9852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:33.514360 9860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.514360 9858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.514438 9852 server_base.cc:1034] running on GCE node
W20260927 02:48:33.514355 9857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.514799 9852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:33.515054 9852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:33.516207 9852 hybrid_clock.cc:648] HybridClock initialized: now 1790477313516195 us; error 29 us; skew 500 ppm
I20260927 02:48:33.517386 9852 webserver.cc:460] Webserver started at http://127.0.98.2:41717/ using document root <none> and password file <none>
I20260927 02:48:33.517589 9852 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:33.517665 9852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:33.517789 9852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:33.518895 9852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "e88c138c86f1491ab19d1bd7de0746c8"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "41a7afde4be257705faae95b644f27c5"
server_key_iv: "4f084d5bf607d4f94d1f4e6da5118184"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.519236 9852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e88c138c86f1491ab19d1bd7de0746c8"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "41a7afde4be257705faae95b644f27c5"
server_key_iv: "4f084d5bf607d4f94d1f4e6da5118184"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.520426 9852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:33.521056 9866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.521202 9852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:33.521281 9852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "e88c138c86f1491ab19d1bd7de0746c8"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "41a7afde4be257705faae95b644f27c5"
server_key_iv: "4f084d5bf607d4f94d1f4e6da5118184"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.521338 9852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:33.534364 9852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:33.534647 9852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:33.534767 9852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:33.534982 9852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:33.535292 9852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:33.535329 9852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.535356 9852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:33.535373 9852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.540732 9852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34325
I20260927 02:48:33.540771 9979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34325 every 8 connection(s)
I20260927 02:48:33.541069 9852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:33.542533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9852
I20260927 02:48:33.542620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:33.542796 392 external_mini_cluster.cc:1428] Setting key 6b8d85f461c87d5a7580c3714e650def
I20260927 02:48:33.543668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40309
--builtin_ntp_servers=127.0.98.20:35325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 02:48:33.545958 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.541721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34289 (local address 127.0.98.62:40309)
0927 02:48:33.541782 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.541784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.542005 (+ 221us) server_negotiation.cc:407] Connection header received
0927 02:48:33.542479 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.542480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.542493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.542564 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.543285 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.543801 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.544741 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.544934 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.544986 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.545713 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.545714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.545715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.545719 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.545733 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.545736 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.545780 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.545857 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.545882 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:48:33.546159 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.541720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40309 (local address 127.0.98.2:34289)
0927 02:48:33.541895 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:33.541935 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:33.542394 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:33.542628 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:33.542633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:33.542644 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:33.543205 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.543210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.543872 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.543873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.544662 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.544664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.544998 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.544999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.545126 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.545590 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:33.545591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:33.545617 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:33.545638 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:33.545786 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:33.545791 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:33.545794 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:33.545856 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 02:48:33.545965 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260927 02:48:33.546583 9980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40309
I20260927 02:48:33.546674 9980 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:33.546893 9980 heartbeater.cc:511] Master 127.0.98.62:40309 requested a full tablet report, sending...
I20260927 02:48:33.547255 9661 ts_manager.cc:194] Registered new tserver with Master: e88c138c86f1491ab19d1bd7de0746c8 (127.0.98.2:34325)
I20260927 02:48:33.547737 9661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34289
W20260927 02:48:33.622637 9983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:33.622898 9983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:33.622922 9983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:33.624087 9983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:33.624123 9983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:33.624146 9983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:33.625452 9983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:33.625631 9983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:33.625875 9983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:33.630630 9991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.630617 9989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.630708 9983 server_base.cc:1034] running on GCE node
W20260927 02:48:33.630617 9988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.631071 9983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:33.631300 9983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:33.632428 9983 hybrid_clock.cc:648] HybridClock initialized: now 1790477313632417 us; error 28 us; skew 500 ppm
I20260927 02:48:33.633590 9983 webserver.cc:460] Webserver started at http://127.0.98.3:43271/ using document root <none> and password file <none>
I20260927 02:48:33.633791 9983 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:33.633839 9983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:33.633960 9983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:33.635026 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a39be6cf70d419a810dd1feb0fcf377"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "75533a537e7f968f8fac9ae0519359cd"
server_key_iv: "16946649e346fb0fea76d6e3ee0bb94c"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.635350 9983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a39be6cf70d419a810dd1feb0fcf377"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "75533a537e7f968f8fac9ae0519359cd"
server_key_iv: "16946649e346fb0fea76d6e3ee0bb94c"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.636487 9983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:33.637163 9997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.637312 9983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:33.637389 9983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "1a39be6cf70d419a810dd1feb0fcf377"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "75533a537e7f968f8fac9ae0519359cd"
server_key_iv: "16946649e346fb0fea76d6e3ee0bb94c"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.637444 9983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:33.652374 9983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:33.652652 9983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:33.652772 9983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:33.652971 9983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:33.653278 9983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:33.653311 9983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.653338 9983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:33.653362 9983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.658680 9983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34129
I20260927 02:48:33.658737 10110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34129 every 8 connection(s)
I20260927 02:48:33.659080 9983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:33.664130 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.659743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48785 (local address 127.0.98.62:40309)
0927 02:48:33.659798 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.659800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.660014 (+ 214us) server_negotiation.cc:407] Connection header received
0927 02:48:33.660523 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.660524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.660537 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.660591 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.661261 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.661745 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.662742 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.662962 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.663025 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.663824 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.663825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.663825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.663830 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.663846 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.663850 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.663894 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.663971 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.663997 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:48:33.664287 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.659711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40309 (local address 127.0.98.3:48785)
0927 02:48:33.659896 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:33.659928 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:33.660420 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:33.660662 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:33.660667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:33.660677 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:33.661168 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.661173 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.661835 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.661836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.662651 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:33.662653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:33.663041 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:33.663042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:33.663179 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:33.663676 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:33.663677 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:33.663706 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:33.663729 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:33.663901 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:33.663907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:33.663910 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:33.663970 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:48:33.664075 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260927 02:48:33.664731 10111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40309
I20260927 02:48:33.664812 10111 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:33.664976 10111 heartbeater.cc:511] Master 127.0.98.62:40309 requested a full tablet report, sending...
I20260927 02:48:33.665285 9661 ts_manager.cc:194] Registered new tserver with Master: 1a39be6cf70d419a810dd1feb0fcf377 (127.0.98.3:34129)
I20260927 02:48:33.665606 9661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48785
I20260927 02:48:33.668251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 9983
I20260927 02:48:33.668329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:33.668462 392 external_mini_cluster.cc:1428] Setting key 5f7910795455bca5a586b0ca7bb973e7
I20260927 02:48:33.669281 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:48:33.670845 9700 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:33.670411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.0.98.62:40309)
0927 02:48:33.670463 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.670465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.670552 (+ 87us) server_negotiation.cc:407] Connection header received
0927 02:48:33.670609 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.670610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.670618 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.670662 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.670760 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50278: BlockingRecv error: recv got EOF from 127.0.0.1:50278 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:48:33.670931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9721
I20260927 02:48:33.675498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9852
I20260927 02:48:33.679105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9983
I20260927 02:48:33.682940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 9630
2026-09-27T02:48:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (599 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T02:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:33Z Disabled control of system clock
I20260927 02:48:33.707202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43571
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:48:33.783517 10122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:33.783715 10122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:33.783733 10122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:33.784806 10122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:33.784839 10122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:33.784852 10122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:33.784861 10122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:33.784875 10122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:33.786134 10122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:33.786320 10122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:33.786549 10122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:33.791106 10127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.791133 10130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.791323 10128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.791186 10122 server_base.cc:1034] running on GCE node
I20260927 02:48:33.791558 10122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:33.791827 10122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:33.792965 10122 hybrid_clock.cc:648] HybridClock initialized: now 1790477313792944 us; error 32 us; skew 500 ppm
I20260927 02:48:33.794149 10122 webserver.cc:460] Webserver started at http://127.0.98.62:38429/ using document root <none> and password file <none>
I20260927 02:48:33.794368 10122 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:33.794414 10122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:33.794543 10122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:33.795600 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "16dcb021dca14a81af0db81e1bc142cc"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "0c3ff759331ace91a63b1af0539b8afb"
server_key_iv: "40f5aa5792110170ddb6ead832a8319c"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.795923 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "16dcb021dca14a81af0db81e1bc142cc"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "0c3ff759331ace91a63b1af0539b8afb"
server_key_iv: "40f5aa5792110170ddb6ead832a8319c"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.797072 10122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:48:33.797694 10136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.797843 10122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:33.797921 10122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "16dcb021dca14a81af0db81e1bc142cc"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "0c3ff759331ace91a63b1af0539b8afb"
server_key_iv: "40f5aa5792110170ddb6ead832a8319c"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.797976 10122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:33.801218 10122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:33.801458 10122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:33.801576 10122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:33.805050 10122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43571
I20260927 02:48:33.805078 10188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43571 every 8 connection(s)
I20260927 02:48:33.805384 10122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:33.805959 10189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:33.808677 10189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc: Bootstrap starting.
I20260927 02:48:33.809198 10189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:33.809423 10189 log.cc:824] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:33.809999 10189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc: No bootstrap required, opened a new log
I20260927 02:48:33.810573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10122
I20260927 02:48:33.810649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:33.810842 392 external_mini_cluster.cc:1428] Setting key 2615dd731930e4bb8c1130da79b1a0d1
I20260927 02:48:33.811973 10189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dcb021dca14a81af0db81e1bc142cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43571 } }
I20260927 02:48:33.812140 10189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:33.812192 10189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16dcb021dca14a81af0db81e1bc142cc, State: Initialized, Role: FOLLOWER
I20260927 02:48:33.812331 10189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dcb021dca14a81af0db81e1bc142cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43571 } }
I20260927 02:48:33.812414 10189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:33.812443 10189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:33.812485 10189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:33.813120 10189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dcb021dca14a81af0db81e1bc142cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43571 } }
I20260927 02:48:33.813179 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.811320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.0.98.62:43571)
0927 02:48:33.811528 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.811538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.811571 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:48:33.811942 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.811951 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.812002 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.812152 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.812771 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.812774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.812774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.812778 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.812815 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.812825 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.812878 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.812953 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.813001 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260927 02:48:33.813215 10189 leader_election.cc:302] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16dcb021dca14a81af0db81e1bc142cc; no voters:
I20260927 02:48:33.813364 10189 leader_election.cc:288] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:33.813618 10189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:33.813648 10194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:33.813761 10194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [term 1 LEADER]: Becoming Leader. State: Replica: 16dcb021dca14a81af0db81e1bc142cc, State: Running, Role: LEADER
I20260927 02:48:33.813840 10194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dcb021dca14a81af0db81e1bc142cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43571 } }
I20260927 02:48:33.814149 10194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16dcb021dca14a81af0db81e1bc142cc. Latest consensus state: current_term: 1 leader_uuid: "16dcb021dca14a81af0db81e1bc142cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dcb021dca14a81af0db81e1bc142cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43571 } } }
I20260927 02:48:33.814157 10195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16dcb021dca14a81af0db81e1bc142cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16dcb021dca14a81af0db81e1bc142cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43571 } } }
I20260927 02:48:33.814275 10195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:33.814229 10194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:33.814476 10203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:33.814990 10203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:33.816371 10203 catalog_manager.cc:1347] Generated new cluster ID: 659a128016a44399899e88316169025a
I20260927 02:48:33.816424 10203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:33.823527 10203 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:33.823689 10203 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:33.827068 10203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16dcb021dca14a81af0db81e1bc142cc: Generated new TSK 0
I20260927 02:48:33.827243 10203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:48:33.829106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43571
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:48:33.906565 10213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:33.906774 10213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:33.906798 10213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:33.907980 10213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:33.908039 10213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:33.908066 10213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:33.909415 10213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:33.909632 10213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:33.909919 10213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:33.915530 10219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.915654 10213 server_base.cc:1034] running on GCE node
W20260927 02:48:33.915505 10218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:33.915517 10221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:33.915947 10213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:33.916209 10213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:33.917367 10213 hybrid_clock.cc:648] HybridClock initialized: now 1790477313917353 us; error 26 us; skew 500 ppm
I20260927 02:48:33.918748 10213 webserver.cc:460] Webserver started at http://127.0.98.1:39497/ using document root <none> and password file <none>
I20260927 02:48:33.919036 10213 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:33.919106 10213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:33.919240 10213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:33.920608 10213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "a508daea9baf46e49979b0b853ba4401"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "04123a3b317d2e36cb281129548192ab"
server_key_iv: "1d705719d3ab114aca261b374422c2f7"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.921017 10213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a508daea9baf46e49979b0b853ba4401"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "04123a3b317d2e36cb281129548192ab"
server_key_iv: "1d705719d3ab114aca261b374422c2f7"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.922634 10213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260927 02:48:33.923527 10227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.923681 10213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:48:33.923753 10213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "a508daea9baf46e49979b0b853ba4401"
format_stamp: "Formatted at 2026-09-27 02:48:33 on dist-test-slave-f4rg"
server_key: "04123a3b317d2e36cb281129548192ab"
server_key_iv: "1d705719d3ab114aca261b374422c2f7"
server_key_version: "encryptionkey@0"
I20260927 02:48:33.923818 10213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:33.928701 10213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:33.928989 10213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:33.929140 10213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:33.929342 10213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:33.929652 10213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:33.929689 10213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.929716 10213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:33.929736 10213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:33.935760 10213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39759
I20260927 02:48:33.936098 10213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:48:33.938522 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.936980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35049 (local address 127.0.98.62:43571)
0927 02:48:33.937021 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:33.937023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:33.937122 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:48:33.937582 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:33.937583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:33.937589 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:33.937655 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:33.938323 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:33.938324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:33.938324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:33.938327 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:33.938339 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:33.938343 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:33.938392 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:33.938438 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 02:48:33.938450 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:48:33.938918 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:33.936761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43571 (local address 127.0.98.1:35049)
0927 02:48:33.936933 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:33.936966 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:33.937497 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:33.937711 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:33.937714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:33.937722 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:33.938201 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:33.938201 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:33.938230 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:33.938260 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:33.938391 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:33.938394 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:33.938396 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:48:33.938445 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:48:33.938492 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 02:48:33.939402 10341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43571
I20260927 02:48:33.939492 10341 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:33.939634 10341 heartbeater.cc:511] Master 127.0.98.62:43571 requested a full tablet report, sending...
I20260927 02:48:33.940109 10153 ts_manager.cc:194] Registered new tserver with Master: a508daea9baf46e49979b0b853ba4401 (127.0.98.1:39759)
I20260927 02:48:33.942632 10340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39759 every 8 connection(s)
I20260927 02:48:33.943646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10213
I20260927 02:48:33.943732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:33.943899 392 external_mini_cluster.cc:1428] Setting key 2e3810111b57041ce1023b037eabb881
I20260927 02:48:33.944697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43571
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 02:48:34.021773 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:34.021986 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:34.022006 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:34.023145 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:34.023180 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:34.023202 10344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:34.024475 10344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:34.024664 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:34.024910 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:34.029613 10350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:34.029618 10349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:34.029618 10352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:34.029942 10344 server_base.cc:1034] running on GCE node
I20260927 02:48:34.030120 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:34.030337 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:34.031551 10344 hybrid_clock.cc:648] HybridClock initialized: now 1790477314031542 us; error 35 us; skew 500 ppm
I20260927 02:48:34.032801 10344 webserver.cc:460] Webserver started at http://127.0.98.2:36769/ using document root <none> and password file <none>
I20260927 02:48:34.033025 10344 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:34.033118 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:34.033246 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:34.034273 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "7a8cd72fc847479db672507950208b83"
format_stamp: "Formatted at 2026-09-27 02:48:34 on dist-test-slave-f4rg"
server_key: "0314eaf6419f4a11c04205d6bdc2cb8c"
server_key_iv: "efc18c35eba0b580edd81c12dfd26fde"
server_key_version: "encryptionkey@0"
I20260927 02:48:34.034593 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7a8cd72fc847479db672507950208b83"
format_stamp: "Formatted at 2026-09-27 02:48:34 on dist-test-slave-f4rg"
server_key: "0314eaf6419f4a11c04205d6bdc2cb8c"
server_key_iv: "efc18c35eba0b580edd81c12dfd26fde"
server_key_version: "encryptionkey@0"
I20260927 02:48:34.035813 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:48:34.036587 10358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:34.036751 10344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:34.036826 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "7a8cd72fc847479db672507950208b83"
format_stamp: "Formatted at 2026-09-27 02:48:34 on dist-test-slave-f4rg"
server_key: "0314eaf6419f4a11c04205d6bdc2cb8c"
server_key_iv: "efc18c35eba0b580edd81c12dfd26fde"
server_key_version: "encryptionkey@0"
I20260927 02:48:34.036890 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:34.040169 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:34.040393 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:34.040504 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:34.040692 10344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:34.040967 10344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:34.041004 10344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:34.041028 10344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:34.041052 10344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:34.046482 10344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33097
I20260927 02:48:34.046546 10471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33097 every 8 connection(s)
I20260927 02:48:34.046841 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:34.048722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10344
I20260927 02:48:34.048802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:34.048996 392 external_mini_cluster.cc:1428] Setting key 293ec0dc6bb5603bea682ffc97e8e1a6
I20260927 02:48:34.049741 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:34.047479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42001 (local address 127.0.98.62:43571)
0927 02:48:34.047535 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:34.047537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:34.047742 (+ 205us) server_negotiation.cc:407] Connection header received
0927 02:48:34.048243 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:34.048244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:34.048250 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:34.048302 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:34.049496 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:34.049497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:34.049497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:34.049501 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:34.049523 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:34.049527 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:34.049571 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:34.049656 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:48:34.049672 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:48:34.049912 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:34.047472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43571 (local address 127.0.98.2:42001)
0927 02:48:34.047629 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:34.047660 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:34.048162 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:34.048351 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:34.048354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:34.048362 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:34.049344 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:34.049345 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:34.049380 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:34.049411 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:34.049576 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:34.049580 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:34.049583 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:48:34.049655 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:48:34.049715 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260927 02:48:34.049962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43571
--builtin_ntp_servers=127.0.98.20:38905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 02:48:34.050319 10472 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43571
I20260927 02:48:34.050395 10472 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:34.050526 10472 heartbeater.cc:511] Master 127.0.98.62:43571 requested a full tablet report, sending...
I20260927 02:48:34.050850 10153 ts_manager.cc:194] Registered new tserver with Master: 7a8cd72fc847479db672507950208b83 (127.0.98.2:33097)
W20260927 02:48:34.129696 10475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:34.129895 10475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:34.129915 10475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:34.131062 10475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:34.131100 10475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:34.131124 10475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:34.132411 10475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:34.132597 10475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:34.132831 10475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:34.137449 10480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:34.137465 10475 server_base.cc:1034] running on GCE node
W20260927 02:48:34.137422 10481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:34.137436 10483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:34.137758 10475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:34.137984 10475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:34.139113 10475 hybrid_clock.cc:648] HybridClock initialized: now 1790477314139100 us; error 29 us; skew 500 ppm
I20260927 02:48:34.140195 10475 webserver.cc:460] Webserver started at http://127.0.98.3:37311/ using document root <none> and password file <none>
I20260927 02:48:34.140447 10475 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:34.140499 10475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:34.140622 10475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:34.141664 10475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "7979e2ac443a42768ecfbdf066499868"
format_stamp: "Formatted at 2026-09-27 02:48:34 on dist-test-slave-f4rg"
server_key: "7de24b3b81a4b92fdd04dfe40e7ee8fb"
server_key_iv: "e33f379f969bbb35da4a3972832587d6"
server_key_version: "encryptionkey@0"
I20260927 02:48:34.141973 10475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7979e2ac443a42768ecfbdf066499868"
format_stamp: "Formatted at 2026-09-27 02:48:34 on dist-test-slave-f4rg"
server_key: "7de24b3b81a4b92fdd04dfe40e7ee8fb"
server_key_iv: "e33f379f969bbb35da4a3972832587d6"
server_key_version: "encryptionkey@0"
I20260927 02:48:34.143147 10475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:48:34.143781 10489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:34.143932 10475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:34.144008 10475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "7979e2ac443a42768ecfbdf066499868"
format_stamp: "Formatted at 2026-09-27 02:48:34 on dist-test-slave-f4rg"
server_key: "7de24b3b81a4b92fdd04dfe40e7ee8fb"
server_key_iv: "e33f379f969bbb35da4a3972832587d6"
server_key_version: "encryptionkey@0"
I20260927 02:48:34.144068 10475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:34.147226 10475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:34.147454 10475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:34.147565 10475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:34.147740 10475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:34.148025 10475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:34.148056 10475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:34.148082 10475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:34.148104 10475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:34.153370 10475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37643
I20260927 02:48:34.153424 10602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37643 every 8 connection(s)
I20260927 02:48:34.153707 10475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:34.154381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10475
I20260927 02:48:34.154469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:34.154665 392 external_mini_cluster.cc:1428] Setting key 57c86111ab8e9305f72ef5ce2454c2d1
I20260927 02:48:34.156487 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:34.154391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39187 (local address 127.0.98.62:43571)
0927 02:48:34.154454 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:34.154456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:34.154694 (+ 238us) server_negotiation.cc:407] Connection header received
0927 02:48:34.155213 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:34.155214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:34.155220 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:34.155261 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:34.156307 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:34.156308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:34.156308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:48:34.156312 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:34.156326 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:48:34.156330 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:34.156374 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:34.156425 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 02:48:34.156436 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:48:34.156656 10605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:34.154353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43571 (local address 127.0.98.3:39187)
0927 02:48:34.154554 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:34.154599 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:34.155127 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:34.155330 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:34.155333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:34.155341 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:34.156181 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:48:34.156181 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:34.156208 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:48:34.156237 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:34.156378 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:34.156381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:34.156383 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:48:34.156433 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:48:34.156479 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260927 02:48:34.157002 10603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43571
I20260927 02:48:34.157088 10603 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:34.157222 10603 heartbeater.cc:511] Master 127.0.98.62:43571 requested a full tablet report, sending...
I20260927 02:48:34.157495 10153 ts_manager.cc:194] Registered new tserver with Master: 7979e2ac443a42768ecfbdf066499868 (127.0.98.3:37643)
I20260927 02:48:34.166337 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 02:48:34.168128 10193 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:48:34.167633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.0.98.62:43571)
0927 02:48:34.167697 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:34.167700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:34.167803 (+ 103us) server_negotiation.cc:407] Connection header received
0927 02:48:34.167870 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:34.167871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:34.167876 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:34.167923 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:34.168059 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58156: BlockingRecv error: recv got EOF from 127.0.0.1:58156 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:48:34.168161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10213
I20260927 02:48:34.172132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10344
I20260927 02:48:34.175889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10475
I20260927 02:48:34.179368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10122
2026-09-27T02:48:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (496 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:34 dist-test-slave-f4rg krb5kdc[10613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:34 dist-test-slave-f4rg krb5kdc[10613](info): set up 2 sockets
Sep 27 02:48:34 dist-test-slave-f4rg krb5kdc[10613](info): commencing operation
krb5kdc: starting...
W20260927 02:48:35.126917 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:48:35.239802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:35.317337 10626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:35.317548 10626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:35.317579 10626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:35.318707 10626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:48:35.318745 10626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:35.318763 10626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:35.318778 10626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:48:35.318794 10626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:48:35.320247 10626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:35.320474 10626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:35.320670 10626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:48:35.325101 10626 server_base.cc:1034] running on GCE node
W20260927 02:48:35.325095 10634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:35.325076 10632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:35.325073 10631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.325438 10626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:35.325682 10626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:35.326845 10626 hybrid_clock.cc:648] HybridClock initialized: now 1790477315326833 us; error 34 us; skew 500 ppm
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:35.329195 10626 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:48:35.329562 10626 webserver.cc:460] Webserver started at http://127.0.98.62:40399/ using document root <none> and password file <none>
I20260927 02:48:35.329736 10626 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:35.329773 10626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:35.329864 10626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:35.330885 10626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "2e908930e97b4966b82a0b9f5aa3d310"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "4161b46e7063472a02197b18df18acd1"
server_key_iv: "9800d14c9e57cd6e3726cfe6754b6584"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.331199 10626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e908930e97b4966b82a0b9f5aa3d310"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "4161b46e7063472a02197b18df18acd1"
server_key_iv: "9800d14c9e57cd6e3726cfe6754b6584"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.332383 10626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:35.333081 10641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.333273 10626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:35.333352 10626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "2e908930e97b4966b82a0b9f5aa3d310"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "4161b46e7063472a02197b18df18acd1"
server_key_iv: "9800d14c9e57cd6e3726cfe6754b6584"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.333421 10626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:35.345043 10626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:35.346837 10626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:35.346969 10626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:35.350505 10626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34827
I20260927 02:48:35.350556 10693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34827 every 8 connection(s)
I20260927 02:48:35.350878 10626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:48:35.351805 10694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:35.353930 10694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310: Bootstrap starting.
I20260927 02:48:35.354202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10626
I20260927 02:48:35.354300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:48:35.354482 392 external_mini_cluster.cc:1428] Setting key 6b4b9e445a496d0028335132f53286fb
I20260927 02:48:35.354591 10694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:35.354882 10694 log.cc:824] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:35.355576 10694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310: No bootstrap required, opened a new log
I20260927 02:48:35.357543 10694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e908930e97b4966b82a0b9f5aa3d310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 02:48:35.357678 10694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:35.357709 10694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e908930e97b4966b82a0b9f5aa3d310, State: Initialized, Role: FOLLOWER
I20260927 02:48:35.357828 10694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e908930e97b4966b82a0b9f5aa3d310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 02:48:35.357906 10694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:48:35.357936 10694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:48:35.357981 10694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:35.358467 10694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e908930e97b4966b82a0b9f5aa3d310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 02:48:35.358556 10694 leader_election.cc:302] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e908930e97b4966b82a0b9f5aa3d310; no voters:
I20260927 02:48:35.358722 10694 leader_election.cc:288] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:48:35.358758 10699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:35.358901 10699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [term 1 LEADER]: Becoming Leader. State: Replica: 2e908930e97b4966b82a0b9f5aa3d310, State: Running, Role: LEADER
I20260927 02:48:35.358971 10699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e908930e97b4966b82a0b9f5aa3d310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } }
I20260927 02:48:35.358999 10694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:48:35.359297 10700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e908930e97b4966b82a0b9f5aa3d310. Latest consensus state: current_term: 1 leader_uuid: "2e908930e97b4966b82a0b9f5aa3d310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e908930e97b4966b82a0b9f5aa3d310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } } }
I20260927 02:48:35.359376 10700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [sys.catalog]: This master's current role is: LEADER
I20260927 02:48:35.359629 10705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:48:35.360121 10705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:48:35.360347 10701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e908930e97b4966b82a0b9f5aa3d310" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e908930e97b4966b82a0b9f5aa3d310" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34827 } } }
I20260927 02:48:35.360420 10701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310 [sys.catalog]: This master's current role is: LEADER
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:35.361939 10705 catalog_manager.cc:1347] Generated new cluster ID: ce939a43f8834db6bf5e460cdd3d8f8f
I20260927 02:48:35.361989 10705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:48:35.364881 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.355005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.0.98.62:34827)
0927 02:48:35.355206 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.355219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.355243 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:48:35.355660 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.355668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.355730 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.355866 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:35.356436 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.357313 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.358147 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.358548 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.358639 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.361123 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:35.361127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:35.361128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:35.361140 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:35.362982 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.363380 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.363383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.363384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.363480 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.363675 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.363676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.363676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.363964 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:35.364117 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.364544 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.364677 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260927 02:48:35.372810 10705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:48:35.373520 10705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:48:35.385483 10705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310: Generated new TSK 0
I20260927 02:48:35.385694 10705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:48:35.413662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34827
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:35.492784 10720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:35.492993 10720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:35.493014 10720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:35.494122 10720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:35.494156 10720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:35.494180 10720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:48:35.495551 10720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:35.495733 10720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:35.495908 10720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:35.500433 10726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:35.500526 10728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:35.500572 10725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.500810 10720 server_base.cc:1034] running on GCE node
I20260927 02:48:35.500985 10720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:35.501181 10720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:35.502303 10720 hybrid_clock.cc:648] HybridClock initialized: now 1790477315502296 us; error 27 us; skew 500 ppm
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:35.504696 10720 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:48:35.505158 10720 webserver.cc:460] Webserver started at http://127.0.98.1:42545/ using document root <none> and password file <none>
I20260927 02:48:35.505352 10720 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:35.505395 10720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:35.505502 10720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:35.506515 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "5967e8bc38ee4568b92f77816235bfe4"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "85c663b1a1f6c376dda0a89d9f1fb366"
server_key_iv: "0d21d0d4c509e77b0fcc96077840a0aa"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.506902 10720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5967e8bc38ee4568b92f77816235bfe4"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "85c663b1a1f6c376dda0a89d9f1fb366"
server_key_iv: "0d21d0d4c509e77b0fcc96077840a0aa"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.508229 10720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:35.508913 10735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.509053 10720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:35.509141 10720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "5967e8bc38ee4568b92f77816235bfe4"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "85c663b1a1f6c376dda0a89d9f1fb366"
server_key_iv: "0d21d0d4c509e77b0fcc96077840a0aa"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.509207 10720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:35.520653 10720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:35.522596 10720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:35.522719 10720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:35.522954 10720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:35.523250 10720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:35.523283 10720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.523303 10720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:35.523325 10720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.528653 10720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36147
I20260927 02:48:35.528692 10848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36147 every 8 connection(s)
I20260927 02:48:35.528991 10720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:35.538195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10720
I20260927 02:48:35.538280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:48:35.538467 392 external_mini_cluster.cc:1428] Setting key afec499b8bdce95cf78a82b7b535994c
I20260927 02:48:35.542645 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.529664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49189 (local address 127.0.98.62:34827)
0927 02:48:35.529722 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.529724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.529906 (+ 182us) server_negotiation.cc:407] Connection header received
0927 02:48:35.530386 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.530387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.530397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.530456 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:35.531407 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.531913 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.532959 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.533163 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.533214 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.535346 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:35.535347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:35.535347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:35.535354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:35.541300 (+ 5946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.541696 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.541698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.541698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.541754 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.541977 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.541978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.541978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.542089 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:35.542145 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.542469 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.542555 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:48:35.542850 10851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.529647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34827 (local address 127.0.98.1:49189)
0927 02:48:35.529796 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.529832 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.530277 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.530518 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.530521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.531112 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:35.531313 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.531317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.532028 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.532029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.532881 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.532883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.533225 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.533226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.533325 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.533866 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:35.533874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:35.535269 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:35.541375 (+ 6106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:35.541382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:35.541384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:35.541616 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:35.541812 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:35.541813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:35.541814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:35.541921 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:35.542142 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:35.542149 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:35.542348 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.542440 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.542605 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260927 02:48:35.543279 10849 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34827
I20260927 02:48:35.543368 10849 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:35.543542 10849 heartbeater.cc:511] Master 127.0.98.62:34827 requested a full tablet report, sending...
I20260927 02:48:35.544030 10658 ts_manager.cc:194] Registered new tserver with Master: 5967e8bc38ee4568b92f77816235bfe4 (127.0.98.1:36147)
I20260927 02:48:35.544613 10658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49189
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:48:35.562651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34827
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:35.640322 10854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:35.640518 10854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:35.640548 10854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:35.641666 10854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:35.641705 10854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:35.641738 10854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:48:35.643195 10854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:35.643421 10854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:35.643626 10854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:35.648454 10859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:35.648312 10860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.648502 10854 server_base.cc:1034] running on GCE node
W20260927 02:48:35.648495 10862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.648830 10854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:35.649049 10854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:35.650184 10854 hybrid_clock.cc:648] HybridClock initialized: now 1790477315650166 us; error 34 us; skew 500 ppm
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:35.652968 10854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:48:35.653478 10854 webserver.cc:460] Webserver started at http://127.0.98.2:39903/ using document root <none> and password file <none>
I20260927 02:48:35.653728 10854 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:35.653787 10854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:35.653918 10854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:35.655273 10854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "786fbd84cfcb432aa6cc75c2ceb86c95"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "f245a4d9a0e82afebbed5823b743b21b"
server_key_iv: "0899078fc5e9284eb43694065c60e64b"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.655680 10854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "786fbd84cfcb432aa6cc75c2ceb86c95"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "f245a4d9a0e82afebbed5823b743b21b"
server_key_iv: "0899078fc5e9284eb43694065c60e64b"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.657328 10854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:48:35.658190 10869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.658391 10854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:48:35.658478 10854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "786fbd84cfcb432aa6cc75c2ceb86c95"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "f245a4d9a0e82afebbed5823b743b21b"
server_key_iv: "0899078fc5e9284eb43694065c60e64b"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.658550 10854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:35.671977 10854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:35.674571 10854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:35.674717 10854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:35.674995 10854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:35.675375 10854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:35.675426 10854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.675455 10854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:35.675482 10854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.681902 10854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36473
I20260927 02:48:35.681969 10982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36473 every 8 connection(s)
I20260927 02:48:35.682271 10854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:48:35.687121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10854
I20260927 02:48:35.687219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:48:35.687402 392 external_mini_cluster.cc:1428] Setting key d86f8ef38ac200d491c772099d699831
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:35.693063 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.683176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52287 (local address 127.0.98.62:34827)
0927 02:48:35.683225 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.683226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.683531 (+ 305us) server_negotiation.cc:407] Connection header received
0927 02:48:35.684205 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.684206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.684218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.684271 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:35.685492 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.686254 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.687734 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.687939 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.688009 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.690116 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:35.690117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:35.690118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:35.690124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:35.691467 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.691943 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.691945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.691945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.692009 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.692303 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.692304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.692304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.692447 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:35.692492 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.692874 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.692964 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:48:35.693612 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.683213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34827 (local address 127.0.98.2:52287)
0927 02:48:35.683380 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.683430 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.684103 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.684346 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.684351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.685104 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:35.685407 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.685414 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.686342 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.686343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.687658 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.687661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.688018 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.688020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.688163 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.688620 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:35.688628 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:35.690007 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:35.691531 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:35.691538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:35.691541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:35.691844 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:35.692082 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:35.692083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:35.692083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:35.692226 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:35.692537 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:35.692543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:35.692711 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.693199 (+ 488us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.693380 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260927 02:48:35.694166 10983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34827
I20260927 02:48:35.694276 10983 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:35.694478 10983 heartbeater.cc:511] Master 127.0.98.62:34827 requested a full tablet report, sending...
I20260927 02:48:35.694851 10658 ts_manager.cc:194] Registered new tserver with Master: 786fbd84cfcb432aa6cc75c2ceb86c95 (127.0.98.2:36473)
I20260927 02:48:35.695250 10658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52287
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:48:35.713548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34827
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:48:35.793326 10988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:48:35.793540 10988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:48:35.793570 10988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:48:35.794847 10988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:48:35.794890 10988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:48:35.794925 10988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:48:35.796376 10988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:48:35.796622 10988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:48:35.796829 10988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:48:35.801599 10996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.801661 10988 server_base.cc:1034] running on GCE node
W20260927 02:48:35.801614 10994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:48:35.801630 10993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:48:35.802032 10988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:48:35.802266 10988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:48:35.803403 10988 hybrid_clock.cc:648] HybridClock initialized: now 1790477315803388 us; error 30 us; skew 500 ppm
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:48:35.805814 10988 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:48:35.806274 10988 webserver.cc:460] Webserver started at http://127.0.98.3:32949/ using document root <none> and password file <none>
I20260927 02:48:35.806488 10988 fs_manager.cc:359] Metadata directory not provided
I20260927 02:48:35.806543 10988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:48:35.806677 10988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:48:35.807796 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "3912509babaf4c18a1993ab748dd9965"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "fb1e93a1c8aa06b030929b5b36d1881d"
server_key_iv: "6c48a240147ad4dc12db0b0cac7cf05f"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.808143 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3912509babaf4c18a1993ab748dd9965"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "fb1e93a1c8aa06b030929b5b36d1881d"
server_key_iv: "6c48a240147ad4dc12db0b0cac7cf05f"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.809335 10988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:48:35.809973 11003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.810114 10988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:48:35.810191 10988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "3912509babaf4c18a1993ab748dd9965"
format_stamp: "Formatted at 2026-09-27 02:48:35 on dist-test-slave-f4rg"
server_key: "fb1e93a1c8aa06b030929b5b36d1881d"
server_key_iv: "6c48a240147ad4dc12db0b0cac7cf05f"
server_key_version: "encryptionkey@0"
I20260927 02:48:35.810257 10988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:48:35.826289 10988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:48:35.828410 10988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:48:35.828557 10988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:48:35.828760 10988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:48:35.829046 10988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:48:35.829077 10988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.829105 10988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:48:35.829121 10988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:48:35.834589 10988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45441
I20260927 02:48:35.834625 11116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45441 every 8 connection(s)
I20260927 02:48:35.834945 10988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:48:35.838270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 10988
I20260927 02:48:35.838379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:48:35.838546 392 external_mini_cluster.cc:1428] Setting key d134b98be2802c9a1ab8b1711cfba237
Sep 27 02:48:35 dist-test-slave-f4rg krb5kdc[10613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:48:35.843758 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.835607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53367 (local address 127.0.98.62:34827)
0927 02:48:35.835653 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.835655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.835866 (+ 211us) server_negotiation.cc:407] Connection header received
0927 02:48:35.836397 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.836398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.836410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.836453 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:35.837413 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.837914 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.839076 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.839275 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.839325 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.841323 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:35.841325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:35.841325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:35.841332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:35.842512 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.842906 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.842907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.842907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.842961 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.843171 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.843172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.843173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.843284 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:35.843338 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.843588 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.843668 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:48:35.843927 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.835597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34827 (local address 127.0.98.3:53367)
0927 02:48:35.835750 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.835787 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.836294 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.836536 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.836539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.837132 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:48:35.837322 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.837328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.837994 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.837995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.838996 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.838999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.839336 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.839336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.839463 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.839946 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:48:35.839953 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:48:35.841239 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:48:35.842598 (+ 1359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:35.842604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:35.842606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:35.842849 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:35.843013 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:48:35.843014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:48:35.843014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:48:35.843117 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:48:35.843335 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:48:35.843341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:48:35.843475 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.843568 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.843719 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260927 02:48:35.844332 11117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34827
I20260927 02:48:35.844419 11117 heartbeater.cc:464] Registering TS with master...
I20260927 02:48:35.844599 11117 heartbeater.cc:511] Master 127.0.98.62:34827 requested a full tablet report, sending...
I20260927 02:48:35.844924 10658 ts_manager.cc:194] Registered new tserver with Master: 3912509babaf4c18a1993ab748dd9965 (127.0.98.3:45441)
I20260927 02:48:35.845332 10658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53367
I20260927 02:48:35.850075 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:48:35.856621 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.851297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.0.98.62:34827)
0927 02:48:35.851356 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.851358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.851471 (+ 113us) server_negotiation.cc:407] Connection header received
0927 02:48:35.851493 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.851493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.851502 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.851556 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:48:35.852056 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.852508 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.853186 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.853360 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.853404 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.854428 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:48:35.854429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:48:35.854429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:48:35.854433 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:48:35.855664 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.855953 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.855953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.855954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.855998 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:48:35.856164 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:48:35.856165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:48:35.856165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:48:35.856291 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:48:35.856359 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.856504 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.856560 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:48:35.858108 10658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:48:35.859719 10658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:48:35.865690 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.861619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36473 (local address 127.0.0.1:39856)
0927 02:48:35.861844 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.861853 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.861903 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.862401 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.862402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.862404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:35.862443 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.862444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.863712 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.863712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.864475 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.864477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.865396 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.865397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.865475 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.865489 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.865569 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.865618 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":47,"threads_started":1}
I20260927 02:48:35.865713 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.862064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.0.98.2:36473)
0927 02:48:35.862205 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.862217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.862226 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:35.862286 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.862289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.862304 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.862505 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:35.862569 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.863642 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.864702 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.865316 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.865390 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.865448 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.865597 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.865635 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260927 02:48:35.865863 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.861606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45441 (local address 127.0.0.1:43592)
0927 02:48:35.861724 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.861762 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.861820 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.862439 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.862440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.862443 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:35.862496 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.862497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.863554 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.863555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.864881 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.864882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.865681 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.865681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.865734 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.865738 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.865766 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.865802 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":22,"thread_start_us":74,"threads_started":1}
I20260927 02:48:35.866223 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.861858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.0.98.3:45441)
0927 02:48:35.862282 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.862286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.862295 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:48:35.862330 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.862332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.862345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.862565 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:35.862625 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.863463 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.864948 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.865599 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.866033 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.866093 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.866116 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.866152 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":82,"threads_started":1}
I20260927 02:48:35.866267 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.861619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36147 (local address 127.0.0.1:55308)
0927 02:48:35.862011 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.862019 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.862069 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.862248 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.862251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.862253 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:35.862317 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.862318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.863656 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.863657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.864850 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.864852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.865759 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.865760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.866110 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.866115 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.866164 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.866202 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":11,"thread_start_us":47,"threads_started":1}
I20260927 02:48:35.866468 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.861856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.0.98.1:36147)
0927 02:48:35.862018 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.862022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.862064 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:48:35.862131 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.862134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.862149 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.862231 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:35.862401 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.863556 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.864935 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.865612 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.865673 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.865726 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.866354 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.866396 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260927 02:48:35.866577 10917 tablet_service.cc:1467] Processing CreateTablet for tablet a1a0bc25b55b421fbb63fcfff5c6499c (DEFAULT_TABLE table=test-table [id=a6fef013586e4faf8f9ab26e1c4c3f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:35.866899 10917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a0bc25b55b421fbb63fcfff5c6499c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:35.867163 11051 tablet_service.cc:1467] Processing CreateTablet for tablet a1a0bc25b55b421fbb63fcfff5c6499c (DEFAULT_TABLE table=test-table [id=a6fef013586e4faf8f9ab26e1c4c3f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:35.867480 11051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a0bc25b55b421fbb63fcfff5c6499c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:35.869422 11131 tablet_bootstrap.cc:492] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Bootstrap starting.
I20260927 02:48:35.869750 11132 tablet_bootstrap.cc:492] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Bootstrap starting.
I20260927 02:48:35.870196 11131 tablet_bootstrap.cc:654] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:35.870353 11132 tablet_bootstrap.cc:654] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:35.870378 10783 tablet_service.cc:1467] Processing CreateTablet for tablet a1a0bc25b55b421fbb63fcfff5c6499c (DEFAULT_TABLE table=test-table [id=a6fef013586e4faf8f9ab26e1c4c3f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:48:35.870618 10783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a0bc25b55b421fbb63fcfff5c6499c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:48:35.870749 11131 log.cc:824] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:35.870916 11132 log.cc:824] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:35.871482 11131 tablet_bootstrap.cc:492] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: No bootstrap required, opened a new log
I20260927 02:48:35.871526 11131 ts_tablet_manager.cc:1397] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:35.871618 11132 tablet_bootstrap.cc:492] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: No bootstrap required, opened a new log
I20260927 02:48:35.871675 11132 ts_tablet_manager.cc:1397] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:48:35.872771 11131 raft_consensus.cc:348] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.872905 11131 raft_consensus.cc:374] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:35.872925 11131 raft_consensus.cc:729] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786fbd84cfcb432aa6cc75c2ceb86c95, State: Initialized, Role: FOLLOWER
I20260927 02:48:35.873019 11131 consensus_queue.cc:260] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.873276 11131 ts_tablet_manager.cc:1428] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:48:35.873397 10983 heartbeater.cc:503] Master 127.0.98.62:34827 was elected leader, sending a full tablet report...
I20260927 02:48:35.873718 11136 tablet_bootstrap.cc:492] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4: Bootstrap starting.
I20260927 02:48:35.873759 11132 raft_consensus.cc:348] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.873915 11132 raft_consensus.cc:374] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:35.873955 11132 raft_consensus.cc:729] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3912509babaf4c18a1993ab748dd9965, State: Initialized, Role: FOLLOWER
I20260927 02:48:35.874050 11132 consensus_queue.cc:260] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.874293 11117 heartbeater.cc:503] Master 127.0.98.62:34827 was elected leader, sending a full tablet report...
I20260927 02:48:35.874294 11132 ts_tablet_manager.cc:1428] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 02:48:35.874327 11136 tablet_bootstrap.cc:654] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4: Neither blocks nor log segments found. Creating new log.
I20260927 02:48:35.874876 11136 log.cc:824] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:48:35.875499 11136 tablet_bootstrap.cc:492] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4: No bootstrap required, opened a new log
I20260927 02:48:35.875551 11136 ts_tablet_manager.cc:1397] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:48:35.876791 11136 raft_consensus.cc:348] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.876909 11136 raft_consensus.cc:374] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:48:35.876927 11136 raft_consensus.cc:729] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5967e8bc38ee4568b92f77816235bfe4, State: Initialized, Role: FOLLOWER
I20260927 02:48:35.877034 11136 consensus_queue.cc:260] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.877254 11136 ts_tablet_manager.cc:1428] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:48:35.877283 10849 heartbeater.cc:503] Master 127.0.98.62:34827 was elected leader, sending a full tablet report...
W20260927 02:48:35.932796 10984 tablet.cc:2369] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:48:35.936322 11135 raft_consensus.cc:482] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:48:35.936436 11135 raft_consensus.cc:504] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.936834 11135 leader_election.cc:288] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3912509babaf4c18a1993ab748dd9965 (127.0.98.3:45441), 5967e8bc38ee4568b92f77816235bfe4 (127.0.98.1:36147)
I20260927 02:48:35.940543 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.937100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54603 (local address 127.0.98.3:45441)
0927 02:48:35.937160 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.937162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.937169 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:48:35.937195 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.937196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.937214 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.937265 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:35.937431 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.938176 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.939619 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.940184 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.940257 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.940268 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.940433 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.940469 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 02:48:35.940544 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.937042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48357 (local address 127.0.98.1:36147)
0927 02:48:35.937130 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.937132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.937383 (+ 251us) server_negotiation.cc:407] Connection header received
0927 02:48:35.937433 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.937434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.937445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.937493 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:48:35.937580 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.938268 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.939618 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.940184 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.940257 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.940268 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.940433 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.940467 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:48:35.940563 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.937002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45441 (local address 127.0.98.2:54603)
0927 02:48:35.937054 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.937065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.937105 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.937287 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.937289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.937291 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:35.937363 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.937365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.938249 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.938249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.939531 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.939533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.940266 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.940267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.940366 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.940368 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.940423 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.940470 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":56}
I20260927 02:48:35.940568 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.937006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36147 (local address 127.0.98.2:48357)
0927 02:48:35.937308 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.937318 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:48:35.937378 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:48:35.937493 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:48:35.937494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:48:35.937495 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:48:35.937538 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.937539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.938325 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.938326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.939531 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:48:35.939533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.940267 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.940267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:48:35.940363 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.940368 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:48:35.940423 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:48:35.940470 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":147,"thread_start_us":106,"threads_started":1}
I20260927 02:48:35.941041 10803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a0bc25b55b421fbb63fcfff5c6499c" candidate_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5967e8bc38ee4568b92f77816235bfe4" is_pre_election: true
I20260927 02:48:35.941111 11071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a0bc25b55b421fbb63fcfff5c6499c" candidate_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3912509babaf4c18a1993ab748dd9965" is_pre_election: true
I20260927 02:48:35.941228 11071 raft_consensus.cc:2454] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786fbd84cfcb432aa6cc75c2ceb86c95 in term 0.
I20260927 02:48:35.941234 10803 raft_consensus.cc:2454] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 786fbd84cfcb432aa6cc75c2ceb86c95 in term 0.
I20260927 02:48:35.941438 10871 leader_election.cc:302] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5967e8bc38ee4568b92f77816235bfe4, 786fbd84cfcb432aa6cc75c2ceb86c95; no voters:
I20260927 02:48:35.941560 11135 raft_consensus.cc:2781] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:48:35.941618 11135 raft_consensus.cc:482] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:48:35.941633 11135 raft_consensus.cc:3037] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:35.942287 11135 raft_consensus.cc:504] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.942404 11135 leader_election.cc:288] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [CANDIDATE]: Term 1 election: Requested vote from peers 3912509babaf4c18a1993ab748dd9965 (127.0.98.3:45441), 5967e8bc38ee4568b92f77816235bfe4 (127.0.98.1:36147)
I20260927 02:48:35.942525 11071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a0bc25b55b421fbb63fcfff5c6499c" candidate_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3912509babaf4c18a1993ab748dd9965"
I20260927 02:48:35.942534 10803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1a0bc25b55b421fbb63fcfff5c6499c" candidate_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5967e8bc38ee4568b92f77816235bfe4"
I20260927 02:48:35.942586 11071 raft_consensus.cc:3037] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:35.942639 10803 raft_consensus.cc:3037] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:48:35.943320 11071 raft_consensus.cc:2454] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786fbd84cfcb432aa6cc75c2ceb86c95 in term 1.
I20260927 02:48:35.943394 10803 raft_consensus.cc:2454] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 786fbd84cfcb432aa6cc75c2ceb86c95 in term 1.
I20260927 02:48:35.943459 10873 leader_election.cc:302] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3912509babaf4c18a1993ab748dd9965, 786fbd84cfcb432aa6cc75c2ceb86c95; no voters:
I20260927 02:48:35.943536 11135 raft_consensus.cc:2781] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:48:35.943675 11135 raft_consensus.cc:686] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 1 LEADER]: Becoming Leader. State: Replica: 786fbd84cfcb432aa6cc75c2ceb86c95, State: Running, Role: LEADER
I20260927 02:48:35.943756 11135 consensus_queue.cc:237] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } }
I20260927 02:48:35.944552 10658 catalog_manager.cc:5495] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 786fbd84cfcb432aa6cc75c2ceb86c95 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "786fbd84cfcb432aa6cc75c2ceb86c95" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 } health_report { overall_health: UNKNOWN } } }
I20260927 02:48:35.957017 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.954779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.98.2:36473)
0927 02:48:35.954854 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 02:48:35.954857 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:48:35.954867 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:48:35.954905 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:48:35.954906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:48:35.954929 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:48:35.955038 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:48:35.955120 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.955583 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.956296 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:48:35.956500 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:48:35.956553 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:48:35.956695 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:48:35.956801 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:48:35.956871 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:48:35.956896 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:48:35.956917 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 02:48:35.958702 10803 raft_consensus.cc:1261] T a1a0bc25b55b421fbb63fcfff5c6499c P 5967e8bc38ee4568b92f77816235bfe4 [term 1 FOLLOWER]: Refusing update from remote peer 786fbd84cfcb432aa6cc75c2ceb86c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:35.958705 11071 raft_consensus.cc:1261] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 1 FOLLOWER]: Refusing update from remote peer 786fbd84cfcb432aa6cc75c2ceb86c95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:48:35.959079 11141 consensus_queue.cc:1035] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3912509babaf4c18a1993ab748dd9965" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:35.959201 11135 consensus_queue.cc:1035] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5967e8bc38ee4568b92f77816235bfe4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:48:35.961478 11144 mvcc.cc:205] Tried to move back new op lower bound from 7333795086164328448 to 7333795086106152960. Current Snapshot: MvccSnapshot[applied={T|T < 7333795086164328448}]
I20260927 02:48:35.962399 11146 mvcc.cc:205] Tried to move back new op lower bound from 7333795086164328448 to 7333795086106152960. Current Snapshot: MvccSnapshot[applied={T|T < 7333795086164328448}]
I20260927 02:48:35.963630 10657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:35.963713 10657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:48:35.964411 10657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e908930e97b4966b82a0b9f5aa3d310: Sending DeleteTablet for 3 replicas of tablet a1a0bc25b55b421fbb63fcfff5c6499c
I20260927 02:48:35.964762 10783 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0bc25b55b421fbb63fcfff5c6499c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55308
I20260927 02:48:35.964740 11051 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0bc25b55b421fbb63fcfff5c6499c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43592
I20260927 02:48:35.964821 10917 tablet_service.cc:1514] Processing DeleteTablet for tablet a1a0bc25b55b421fbb63fcfff5c6499c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:48:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39856
I20260927 02:48:35.965025 11156 tablet_replica.cc:330] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: stopping tablet replica
I20260927 02:48:35.965137 11156 raft_consensus.cc:2229] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:48:35.965238 11156 raft_consensus.cc:2258] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:35.965319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10720
I20260927 02:48:35.966190 11156 ts_tablet_manager.cc:1905] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:35.967765 11156 ts_tablet_manager.cc:1918] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:35.967830 11156 log.cc:1196] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/a1a0bc25b55b421fbb63fcfff5c6499c
I20260927 02:48:35.968031 11155 tablet_replica.cc:330] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: stopping tablet replica
I20260927 02:48:35.968099 11156 ts_tablet_manager.cc:1939] T a1a0bc25b55b421fbb63fcfff5c6499c P 3912509babaf4c18a1993ab748dd9965: Deleting consensus metadata
I20260927 02:48:35.968223 11155 raft_consensus.cc:2229] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:48:35.968360 11155 raft_consensus.cc:2258] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:48:35.968482 10645 catalog_manager.cc:4841] TS 3912509babaf4c18a1993ab748dd9965 (127.0.98.3:45441): tablet a1a0bc25b55b421fbb63fcfff5c6499c (table test-table [id=a6fef013586e4faf8f9ab26e1c4c3f45]) successfully deleted
I20260927 02:48:35.968818 11155 ts_tablet_manager.cc:1905] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:48:35.970268 11155 ts_tablet_manager.cc:1918] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:48:35.970336 11155 log.cc:1196] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/a1a0bc25b55b421fbb63fcfff5c6499c
I20260927 02:48:35.970593 11155 ts_tablet_manager.cc:1939] T a1a0bc25b55b421fbb63fcfff5c6499c P 786fbd84cfcb432aa6cc75c2ceb86c95: Deleting consensus metadata
I20260927 02:48:35.970953 10644 catalog_manager.cc:4841] TS 786fbd84cfcb432aa6cc75c2ceb86c95 (127.0.98.2:36473): tablet a1a0bc25b55b421fbb63fcfff5c6499c (table test-table [id=a6fef013586e4faf8f9ab26e1c4c3f45]) successfully deleted
W20260927 02:48:35.972225 10643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36147 (error 108)
I20260927 02:48:35.972419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10854
I20260927 02:48:35.973547 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:48:35.972578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36147 (local address 127.0.0.1:55318)
0927 02:48:35.973472 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 02:48:35.973517 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":873}
W20260927 02:48:35.973680 10643 catalog_manager.cc:4584] TS 5967e8bc38ee4568b92f77816235bfe4 (127.0.98.1:36147): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1a0bc25b55b421fbb63fcfff5c6499c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36147: connect: Connection refused (error 111)
I20260927 02:48:35.976821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10988
I20260927 02:48:35.980890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 10626
2026-09-27T02:48:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1802 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:48:36 dist-test-slave-f4rg krb5kdc[11160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:48:36 dist-test-slave-f4rg krb5kdc[11160](info): set up 2 sockets
Sep 27 02:48:36 dist-test-slave-f4rg krb5kdc[11160](info): commencing operation
krb5kdc: starting...
W20260927 02:48:36.931036 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:48:37 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:48:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:48:37.080864 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskc9ab2Y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:48:39.379 UTC [11190] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:48:39.379 UTC [11190] LOG: listening on IPv4 address "127.0.98.20", port 37493
2026-09-27 02:48:39.382 UTC [11190] LOG: listening on Unix socket "/tmp/.s.PGSQL.37493"
2026-09-27 02:48:39.408 UTC [11193] LOG: database system was shut down at 2026-09-27 02:48:38 UTC
2026-09-27 02:48:39.411 UTC [11190] LOG: database system is ready to accept connections
I20260927 02:48:40.278595 392 mini_postgres.cc:96] Postgres bound to 37493
2026-09-27 02:48:40.284 UTC [11201] FATAL: database "slave" does not exist
127.0.98.20:37493 - accepting connections
I20260927 02:48:40.284564 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:48:40.297827 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:48:40.719379 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:48:40.719466 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-admin
2026-09-27 02:48:40,988 [I] DB FLAVOR :POSTGRES
2026-09-27 02:48:40,988 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:48:40,988 [I] Checking connection..
2026-09-27 02:48:40,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:48:41,243 [I] Checking connection passed.
2026-09-27 02:48:41,243 [I] --------- Verifying version history table ---------
2026-09-27 02:48:41,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:48:41,498 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:48:41,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:48:41,746 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:48:41,747 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:48:41,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:48:41.969 UTC [11265] WARNING: there is no transaction in progress
2026-09-27 02:48:41,975 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:48:41,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:48:42,212 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:48:42,212 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:48:42,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:48:42,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:48:42,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f4rg.c.gcp-upstream.internal','N') ;"
2026-09-27 02:48:42,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:48:43,165 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:48:43,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:48:43,392 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:48:43,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:48:43,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:48:43,858 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:48:43,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:48:44.793 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.810 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.825 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.839 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.855 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.874 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.927 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.934 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.948 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.958 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.968 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.981 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.991 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:44.998 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:45.004 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:45.011 UTC [11373] WARNING: there is no transaction in progress
2026-09-27 02:48:45,610 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:48:45,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:48:45,849 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:48:45,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:48:46,102 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:48:46,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:48:46,358 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:48:46,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:48:46,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:48:46,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f4rg.c.gcp-upstream.internal';"
2026-09-27 02:48:47,056 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:48:47,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:48:47,290 [I] DB_PATCHES have already been applied
I20260927 02:48:47.295864 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskc9ab2Y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:48:47.295928 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:48:47.298732 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34351
Sep 27, 2026 2:48:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:48:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:48:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:48:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:48:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:48:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:48:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:48:48.288635 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.004s user 0.008s sys 0.000s
Sep 27, 2026 2:49:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:49:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 2:49:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:49:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:49:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:49:09.433274 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:49:09 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477349, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:49:09.522773 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:49:09.532574 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:49:10.363641 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:49:10.363718 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms
I20260927 02:49:10.363736 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:37493
2026-09-27 02:49:10,523 [I] DB FLAVOR :POSTGRES
2026-09-27 02:49:10,523 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:49:10,523 [I] Checking connection
2026-09-27 02:49:10,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:49:10,778 [I] connection success
2026-09-27 02:49:10,778 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:49:10,778 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:49:10,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:49:11,001 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:49:11,002 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:49:11,002 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:49:11,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37493/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:49:11,270 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:49:11.704972 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:49:11.904739 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:49:11.947168 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:49:11 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477349, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:49:12.327907 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:49:12.328024 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:49:12.328044 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:49:12.331792 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11553
I20260927 02:49:12.331907 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:54627
Sep 27, 2026 2:49:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:49:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:49:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:49:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:49:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:49:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 02:49:14 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477354, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:49:14.379549 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.857s user 0.008s sys 0.135s
I20260927 02:49:14.379654 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:49:14.379720 392 mini_ranger_kms.cc:401] 127.0.98.20:54627/kms/v1/keys
Sep 27 02:49:14 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477354, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:49:15 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477354, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:49:17 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:49:17.318009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38923
--ranger_config_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54627/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:49:17.422276 11604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:49:17.422487 11604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:49:17.422518 11604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:49:17.423710 11604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:49:17.423754 11604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:49:17.423767 11604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:49:17.423779 11604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:49:17.423789 11604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:49:17.425328 11604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:54627/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38923
--ranger_config_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:49:17.425580 11604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:49:17.425797 11604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:49:17.431070 11609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:17.431097 11612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:17.431097 11610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:49:17.431203 11604 server_base.cc:1034] running on GCE node
I20260927 02:49:17.431468 11604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:49:17.431764 11604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:49:17.432904 11604 hybrid_clock.cc:648] HybridClock initialized: now 1790477357432880 us; error 40 us; skew 500 ppm
Sep 27 02:49:17 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:49:17.435354 11604 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:49:17.435793 11604 webserver.cc:460] Webserver started at http://127.0.98.62:35387/ using document root <none> and password file <none>
I20260927 02:49:17.436013 11604 fs_manager.cc:359] Metadata directory not provided
I20260927 02:49:17.436053 11604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:49:17.436177 11604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:49:17 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:49:17.489070 11604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "174338e435434853897e2d23afde8d45"
format_stamp: "Formatted at 2026-09-27 02:49:17 on dist-test-slave-f4rg"
server_key: "9b4a030586afa131eacd8d64cc7fd2c7"
server_key_iv: "3a6469bf7ebc907e83fb9f00657798cb"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:17.489557 11604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "174338e435434853897e2d23afde8d45"
format_stamp: "Formatted at 2026-09-27 02:49:17 on dist-test-slave-f4rg"
server_key: "9b4a030586afa131eacd8d64cc7fd2c7"
server_key_iv: "3a6469bf7ebc907e83fb9f00657798cb"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:17.491252 11604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260927 02:49:17.502985 11619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:17.503202 11604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:49:17.503311 11604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "174338e435434853897e2d23afde8d45"
format_stamp: "Formatted at 2026-09-27 02:49:17 on dist-test-slave-f4rg"
server_key: "9b4a030586afa131eacd8d64cc7fd2c7"
server_key_iv: "3a6469bf7ebc907e83fb9f00657798cb"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:17.503413 11604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:49:17.516402 11604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:49:17.518195 11604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:49:17.518334 11604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:49:17.523111 11604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38923
I20260927 02:49:17.523145 11671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38923 every 8 connection(s)
I20260927 02:49:17.524046 11604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:49:17.524715 11672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:49:17.526943 11672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45: Bootstrap starting.
I20260927 02:49:17.527474 11672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45: Neither blocks nor log segments found. Creating new log.
I20260927 02:49:17.527714 11672 log.cc:824] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45: Log is configured to *not* fsync() on all Append() calls
I20260927 02:49:17.528316 11672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45: No bootstrap required, opened a new log
I20260927 02:49:17.530184 11672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174338e435434853897e2d23afde8d45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 02:49:17.530314 11672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:49:17.530344 11672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 174338e435434853897e2d23afde8d45, State: Initialized, Role: FOLLOWER
I20260927 02:49:17.530479 11672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174338e435434853897e2d23afde8d45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 02:49:17.530552 11672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:49:17.530591 11672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:49:17.530634 11672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:49:17.531170 11672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174338e435434853897e2d23afde8d45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 02:49:17.531262 11672 leader_election.cc:302] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 174338e435434853897e2d23afde8d45; no voters:
I20260927 02:49:17.531409 11672 leader_election.cc:288] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:49:17.531443 11675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:49:17.531579 11675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [term 1 LEADER]: Becoming Leader. State: Replica: 174338e435434853897e2d23afde8d45, State: Running, Role: LEADER
I20260927 02:49:17.531647 11672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:49:17.531678 11675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174338e435434853897e2d23afde8d45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } }
I20260927 02:49:17.532179 11677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 174338e435434853897e2d23afde8d45. Latest consensus state: current_term: 1 leader_uuid: "174338e435434853897e2d23afde8d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174338e435434853897e2d23afde8d45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } }
I20260927 02:49:17.532310 11677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [sys.catalog]: This master's current role is: LEADER
I20260927 02:49:17.532500 11676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "174338e435434853897e2d23afde8d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "174338e435434853897e2d23afde8d45" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } }
I20260927 02:49:17.532593 11676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45 [sys.catalog]: This master's current role is: LEADER
I20260927 02:49:17.532747 11684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:49:17.533018 11672 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:49:17.533215 11684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:49:17.534143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 11604
I20260927 02:49:17.534248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:49:17.534441 392 external_mini_cluster.cc:1428] Setting key b160292fac858b1bc0e7a74ee655f8ed
I20260927 02:49:17.538762 11684 catalog_manager.cc:1347] Generated new cluster ID: a7fb4002ea2545d2bc23a161fd40cef6
I20260927 02:49:17.538894 11684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:49:17.545898 11684 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:49:17.547422 11684 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:49:17.553750 11684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45: Generated new TSK 0
I20260927 02:49:17.554029 11684 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 27 02:49:17 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:49:17.563333 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:17.535914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.0.98.62:38923)
0927 02:49:17.536446 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:17.536459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:17.545761 (+ 9302us) server_negotiation.cc:407] Connection header received
0927 02:49:17.545924 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:17.545933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:17.545998 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:17.546173 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:49:17.550939 (+ 4766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:17.552428 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:17.555438 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:17.556076 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:17.556516 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:17.559003 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:49:17.559008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:49:17.559009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:49:17.559022 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:49:17.561427 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:17.561810 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:17.561813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:17.561814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:17.561916 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:17.562097 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:17.562097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:17.562098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:17.562392 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:49:17.562882 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:17.562956 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:49:17.563099 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:49:18.278891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:49:18.365101 11717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:49:18.365304 11717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:49:18.365337 11717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:49:18.366498 11717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:49:18.366539 11717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:49:18.366572 11717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:49:18.368088 11717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:54627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:49:18.368309 11717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:49:18.368512 11717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:49:18.373477 11722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:18.373543 11723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:18.374053 11725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:49:18.374076 11717 server_base.cc:1034] running on GCE node
I20260927 02:49:18.374372 11717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:49:18.374636 11717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:49:18.375830 11717 hybrid_clock.cc:648] HybridClock initialized: now 1790477358375807 us; error 35 us; skew 500 ppm
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:49:18.378660 11717 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:49:18.379252 11717 webserver.cc:460] Webserver started at http://127.0.98.1:36449/ using document root <none> and password file <none>
I20260927 02:49:18.379537 11717 fs_manager.cc:359] Metadata directory not provided
I20260927 02:49:18.379608 11717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:49:18.379750 11717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:49:18.393402 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "3238ec3768e944e6894b516eaf630195"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "024dbc5ca71b4c1cb632537d1c931ea1"
server_key_iv: "4b32e23d31157d021c565bb941d5b0d5"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.393930 11717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3238ec3768e944e6894b516eaf630195"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "024dbc5ca71b4c1cb632537d1c931ea1"
server_key_iv: "4b32e23d31157d021c565bb941d5b0d5"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.398115 11717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 02:49:18.412854 11732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.413043 11717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:49:18.413130 11717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "3238ec3768e944e6894b516eaf630195"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "024dbc5ca71b4c1cb632537d1c931ea1"
server_key_iv: "4b32e23d31157d021c565bb941d5b0d5"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.413232 11717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:49:18.441783 11717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:49:18.444222 11717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:49:18.444372 11717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:49:18.444623 11717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:49:18.444990 11717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:49:18.445035 11717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.445103 11717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:49:18.445150 11717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.453239 11717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39507
I20260927 02:49:18.453704 11717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:49:18.455552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 11717
I20260927 02:49:18.455654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:49:18.455870 392 external_mini_cluster.cc:1428] Setting key 286796768d3166369c18795736b9348b
I20260927 02:49:18.454605 11845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39507 every 8 connection(s)
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:49:18.465054 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.455201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45605 (local address 127.0.98.62:38923)
0927 02:49:18.456540 (+ 1339us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:18.456542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:18.456549 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:49:18.456569 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:18.456569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:18.456580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:18.456650 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:49:18.457401 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.458114 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.459204 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.459464 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.459525 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.462030 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:49:18.462032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:49:18.462032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:49:18.462041 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:49:18.463759 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.464135 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.464136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.464137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.464205 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.464417 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.464418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.464418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.464562 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:49:18.464624 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:18.464864 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 02:49:18.464945 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":75,"threads_started":1}
I20260927 02:49:18.465106 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.455183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.98.1:45605)
0927 02:49:18.455357 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:18.455418 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:18.456029 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:18.456741 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:18.456746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:18.457004 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:49:18.457288 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.457294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.458205 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.458206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.459131 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.459133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.459590 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.459591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.459740 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.460342 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:49:18.460352 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:49:18.461916 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:49:18.463837 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:49:18.463846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:49:18.463849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:49:18.464066 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:49:18.464264 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:49:18.464264 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:49:18.464265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:49:18.464363 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:49:18.464620 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:49:18.464623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:49:18.464750 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 02:49:18.464825 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:49:18.464943 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260927 02:49:18.465662 11846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38923
I20260927 02:49:18.465770 11846 heartbeater.cc:464] Registering TS with master...
I20260927 02:49:18.465960 11846 heartbeater.cc:511] Master 127.0.98.62:38923 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:49:18.466528 11636 ts_manager.cc:194] Registered new tserver with Master: 3238ec3768e944e6894b516eaf630195 (127.0.98.1:39507)
I20260927 02:49:18.467461 11636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:49:18.484951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:49:18.593637 11854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:49:18.593921 11854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:49:18.593962 11854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:49:18.595727 11854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:49:18.595789 11854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:49:18.595829 11854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:49:18.597951 11854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:54627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:49:18.598188 11854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:49:18.598498 11854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:49:18.604656 11862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:18.604686 11861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:18.604684 11864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:49:18.605708 11854 server_base.cc:1034] running on GCE node
I20260927 02:49:18.605921 11854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:49:18.606182 11854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:49:18.607348 11854 hybrid_clock.cc:648] HybridClock initialized: now 1790477358607329 us; error 36 us; skew 500 ppm
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:49:18.610487 11854 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:49:18.611034 11854 webserver.cc:460] Webserver started at http://127.0.98.2:44653/ using document root <none> and password file <none>
I20260927 02:49:18.611280 11854 fs_manager.cc:359] Metadata directory not provided
I20260927 02:49:18.611340 11854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:49:18.611532 11854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:49:18.625856 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f00bd10da5e4a17ba8c840cc87aedc9"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "c4cbbeccd3d1870982593cbba4202506"
server_key_iv: "37bd04cfc32593f84c7e61bed2b95c99"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.626420 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f00bd10da5e4a17ba8c840cc87aedc9"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "c4cbbeccd3d1870982593cbba4202506"
server_key_iv: "37bd04cfc32593f84c7e61bed2b95c99"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.628670 11854 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:49:18.642294 11871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.642549 11854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:49:18.642635 11854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "6f00bd10da5e4a17ba8c840cc87aedc9"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "c4cbbeccd3d1870982593cbba4202506"
server_key_iv: "37bd04cfc32593f84c7e61bed2b95c99"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.642721 11854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:49:18.676432 11854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:49:18.679764 11854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:49:18.680011 11854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:49:18.680308 11854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:49:18.680734 11854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:49:18.680827 11854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.680913 11854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:49:18.680968 11854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.688704 11854 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44759
I20260927 02:49:18.689121 11854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:49:18.689999 11984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44759 every 8 connection(s)
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:49:18.696902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 11854
I20260927 02:49:18.697013 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:49:18.697212 392 external_mini_cluster.cc:1428] Setting key eee194e6f9fbad23a87316918e0a0f2c
I20260927 02:49:18.703315 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.690151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35783 (local address 127.0.98.62:38923)
0927 02:49:18.690195 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:18.690198 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:18.690414 (+ 216us) server_negotiation.cc:407] Connection header received
0927 02:49:18.691087 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:18.691088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:18.691101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:18.691129 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:49:18.691955 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.692704 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.694144 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.694444 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.694487 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.700163 (+ 5676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:49:18.700165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:49:18.700165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:49:18.700173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:49:18.701812 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.702273 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.702274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.702274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.702325 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.702568 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.702568 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.702569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.702695 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:49:18.702731 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:18.703124 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 02:49:18.703213 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:49:18.704058 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.689854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.98.2:35783)
0927 02:49:18.690266 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:18.690309 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:18.690962 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:18.691318 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:18.691322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:18.691565 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:49:18.691836 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.691843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.692815 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.692815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.694044 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.694048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.694578 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.694579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.694721 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.695365 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:49:18.695374 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:49:18.700041 (+ 4667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:49:18.701906 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:49:18.701914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:49:18.701917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:49:18.702201 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:49:18.702386 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:49:18.702387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:49:18.702387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:49:18.702507 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:49:18.702768 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:49:18.702773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:49:18.702983 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 02:49:18.703640 (+ 657us) client_negotiation.cc:240] Negotiation successful
0927 02:49:18.703802 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":94,"threads_started":1}
I20260927 02:49:18.704660 11985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38923
I20260927 02:49:18.704746 11985 heartbeater.cc:464] Registering TS with master...
I20260927 02:49:18.704933 11985 heartbeater.cc:511] Master 127.0.98.62:38923 requested a full tablet report, sending...
I20260927 02:49:18.705302 11636 ts_manager.cc:194] Registered new tserver with Master: 6f00bd10da5e4a17ba8c840cc87aedc9 (127.0.98.2:44759)
I20260927 02:49:18.705781 11636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35783
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:49:18.723567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:54627/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:49:18.838933 11990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:49:18.839195 11990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:49:18.839226 11990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:49:18.840847 11990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:49:18.840894 11990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:49:18.840930 11990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:49:18.842986 11990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:54627/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:49:18.843210 11990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:49:18.843518 11990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:49:18.850647 11999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:18.851142 11996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:49:18.851379 11997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:49:18.852476 11990 server_base.cc:1034] running on GCE node
I20260927 02:49:18.852775 11990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:49:18.853036 11990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:49:18.860427 11990 hybrid_clock.cc:648] HybridClock initialized: now 1790477358860393 us; error 43 us; skew 500 ppm
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:49:18.863520 11990 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:49:18.864086 11990 webserver.cc:460] Webserver started at http://127.0.98.3:46045/ using document root <none> and password file <none>
I20260927 02:49:18.864331 11990 fs_manager.cc:359] Metadata directory not provided
I20260927 02:49:18.864378 11990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:49:18.864511 11990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:49:18.879137 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "35e89ad078fc427c9f4d859af5091d49"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "e3730b1acec9c15c8baf10d3b4fab103"
server_key_iv: "3df77a6a65e63469a06a6125527e4407"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.879727 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "35e89ad078fc427c9f4d859af5091d49"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "e3730b1acec9c15c8baf10d3b4fab103"
server_key_iv: "3df77a6a65e63469a06a6125527e4407"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.881879 11990 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:49:18.908215 12006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.908529 11990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:49:18.908645 11990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "35e89ad078fc427c9f4d859af5091d49"
format_stamp: "Formatted at 2026-09-27 02:49:18 on dist-test-slave-f4rg"
server_key: "e3730b1acec9c15c8baf10d3b4fab103"
server_key_iv: "3df77a6a65e63469a06a6125527e4407"
server_key_version: "kuduclusterkey@0"
I20260927 02:49:18.908747 11990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:49:18.935245 11990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:49:18.937582 11990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:49:18.937719 11990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:49:18.937960 11990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:49:18.938313 11990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:49:18.938359 11990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.938390 11990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:49:18.938431 11990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:49:18.945160 11990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44195
I20260927 02:49:18.945600 11990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:49:18.947204 12119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44195 every 8 connection(s)
Sep 27 02:49:18 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:49:18.954473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 11990
I20260927 02:49:18.954571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:49:18.954774 392 external_mini_cluster.cc:1428] Setting key c9592130e4e3eb76a1853af99ed09b29
I20260927 02:49:18.957406 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.946327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50917 (local address 127.0.98.62:38923)
0927 02:49:18.946375 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:18.946377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:18.947785 (+ 1408us) server_negotiation.cc:407] Connection header received
0927 02:49:18.948457 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:18.948458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:18.948471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:18.948501 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:49:18.949098 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.949585 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.950556 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.950760 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.950848 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.952857 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:49:18.952859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:49:18.952859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:49:18.952867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:49:18.954174 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.954719 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.954721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.954721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.954777 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.956774 (+ 1997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.956775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.956776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.956892 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:49:18.956949 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:18.957227 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 02:49:18.957307 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:49:18.957577 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.946330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.98.3:50917)
0927 02:49:18.947616 (+ 1286us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:18.947664 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:18.948318 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:18.948647 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:18.948651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:18.948824 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:49:18.949014 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.949018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.949663 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.949664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.950482 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.950484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.950848 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.950849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.950958 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.951515 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:49:18.951524 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:49:18.952776 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:49:18.954244 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:49:18.954251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:49:18.954253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:49:18.954634 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:49:18.956593 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:49:18.956594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:49:18.956594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:49:18.956703 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:49:18.956948 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:49:18.956954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:49:18.957103 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 02:49:18.957185 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:49:18.957351 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":72,"threads_started":1}
I20260927 02:49:18.958076 12120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38923
I20260927 02:49:18.958175 12120 heartbeater.cc:464] Registering TS with master...
I20260927 02:49:18.958352 12120 heartbeater.cc:511] Master 127.0.98.62:38923 requested a full tablet report, sending...
I20260927 02:49:18.958683 11636 ts_manager.cc:194] Registered new tserver with Master: 35e89ad078fc427c9f4d859af5091d49 (127.0.98.3:44195)
I20260927 02:49:18.959096 11636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50917
I20260927 02:49:18.966367 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:49:18.974403 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.967848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.98.62:38923)
0927 02:49:18.967893 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:18.967895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:18.968010 (+ 115us) server_negotiation.cc:407] Connection header received
0927 02:49:18.968070 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:18.968071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:18.968079 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:18.968124 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:49:18.968649 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.969212 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.970265 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.970558 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.970603 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.971953 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:49:18.971954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:49:18.971954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:49:18.971962 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:49:18.973216 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.973573 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.973574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.973574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.973629 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:49:18.973783 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:49:18.973784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:49:18.973784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:49:18.973919 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:49:18.974014 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:18.974221 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 02:49:18.974301 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:49:18.976487 11636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:49:18.978386 11636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:49:18.985785 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.980879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.0.98.2:44759)
0927 02:49:18.981034 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:18.981039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:18.981632 (+ 593us) server_negotiation.cc:407] Connection header received
0927 02:49:18.981774 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:18.981777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:18.981792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:18.981865 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:49:18.981997 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.982710 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.984166 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.985257 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.985304 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.985392 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:18.985649 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:49:18.985692 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260927 02:49:18.986356 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.980620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44759 (local address 127.0.0.1:44560)
0927 02:49:18.981221 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:18.981273 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:18.981704 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:18.981871 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:18.981874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:18.981876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:49:18.981946 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.981948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.982774 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.982776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.984051 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.984054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.985451 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.985452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.985528 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.985544 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:49:18.986206 (+ 662us) client_negotiation.cc:240] Negotiation successful
0927 02:49:18.986274 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":81,"threads_started":1}
I20260927 02:49:18.987327 11918 tablet_service.cc:1467] Processing CreateTablet for tablet 8c879802953b48bda0672b4a757b0991 (DEFAULT_TABLE table=test-table [id=744347527bf7490ebb0fa92c176fa4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:49:18.987747 11918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c879802953b48bda0672b4a757b0991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:49:18.993515 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.980621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39507 (local address 127.0.0.1:59962)
0927 02:49:18.981412 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:18.981649 (+ 237us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:18.981674 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:18.983110 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:18.983111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:18.983113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:49:18.983154 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.983156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.984589 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.984590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.991416 (+ 6826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.991421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.993237 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.993238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.993310 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.993315 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:49:18.993369 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:49:18.993427 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":78,"threads_started":1}
I20260927 02:49:18.994032 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.980882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59962 (local address 127.0.98.1:39507)
0927 02:49:18.982911 (+ 2029us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:18.982916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:18.982925 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:49:18.982957 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:18.982958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:18.982973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:18.983045 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:49:18.983484 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.984507 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.992200 (+ 7693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.993057 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.993134 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.993196 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:18.993914 (+ 718us) server_negotiation.cc:299] Negotiation successful
0927 02:49:18.993960 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1997,"thread_start_us":73,"threads_started":1}
I20260927 02:49:18.994583 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.981100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44195 (local address 127.0.0.1:57724)
0927 02:49:18.981489 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:18.981497 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:18.981551 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:18.986125 (+ 4574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:18.986126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:18.986128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:49:18.986169 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.986171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.992178 (+ 6007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.992179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.993294 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:18.993298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.994352 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.994353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:18.994413 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.994418 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:49:18.994465 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:49:18.994510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":59,"threads_started":1}
I20260927 02:49:18.995036 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:18.981202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57724 (local address 127.0.98.3:44195)
0927 02:49:18.985756 (+ 4554us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:18.985761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:18.985771 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:49:18.985807 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:18.985809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:18.985825 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:18.986030 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:49:18.986244 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.992073 (+ 5829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.993425 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:18.994169 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:18.994236 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:18.994316 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:18.994911 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 02:49:18.994956 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4516,"thread_start_us":89,"threads_started":1}
I20260927 02:49:18.994931 11762 tablet_service.cc:1467] Processing CreateTablet for tablet 8c879802953b48bda0672b4a757b0991 (DEFAULT_TABLE table=test-table [id=744347527bf7490ebb0fa92c176fa4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:49:18.995294 11762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c879802953b48bda0672b4a757b0991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:49:18.999466 12138 tablet_bootstrap.cc:492] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Bootstrap starting.
I20260927 02:49:18.999505 12054 tablet_service.cc:1467] Processing CreateTablet for tablet 8c879802953b48bda0672b4a757b0991 (DEFAULT_TABLE table=test-table [id=744347527bf7490ebb0fa92c176fa4e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:49:18.999905 12054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c879802953b48bda0672b4a757b0991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:49:19.000669 12138 tablet_bootstrap.cc:654] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Neither blocks nor log segments found. Creating new log.
I20260927 02:49:19.001053 12138 log.cc:824] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:49:19.001899 12138 tablet_bootstrap.cc:492] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: No bootstrap required, opened a new log
I20260927 02:49:19.001969 12138 ts_tablet_manager.cc:1397] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 02:49:19.004014 12138 raft_consensus.cc:348] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.004180 12138 raft_consensus.cc:374] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:49:19.004212 12138 raft_consensus.cc:729] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f00bd10da5e4a17ba8c840cc87aedc9, State: Initialized, Role: FOLLOWER
I20260927 02:49:19.004331 12138 consensus_queue.cc:260] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.004652 12138 ts_tablet_manager.cc:1428] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:49:19.004774 11985 heartbeater.cc:503] Master 127.0.98.62:38923 was elected leader, sending a full tablet report...
I20260927 02:49:19.005769 12139 tablet_bootstrap.cc:492] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195: Bootstrap starting.
I20260927 02:49:19.006457 12139 tablet_bootstrap.cc:654] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195: Neither blocks nor log segments found. Creating new log.
I20260927 02:49:19.007256 12139 log.cc:824] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195: Log is configured to *not* fsync() on all Append() calls
I20260927 02:49:19.007767 12141 tablet_bootstrap.cc:492] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49: Bootstrap starting.
I20260927 02:49:19.008857 12141 tablet_bootstrap.cc:654] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49: Neither blocks nor log segments found. Creating new log.
I20260927 02:49:19.009187 12141 log.cc:824] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49: Log is configured to *not* fsync() on all Append() calls
I20260927 02:49:19.009644 12139 tablet_bootstrap.cc:492] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195: No bootstrap required, opened a new log
I20260927 02:49:19.009723 12139 ts_tablet_manager.cc:1397] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 02:49:19.012722 12139 raft_consensus.cc:348] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.012972 12139 raft_consensus.cc:374] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:49:19.013052 12139 raft_consensus.cc:729] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3238ec3768e944e6894b516eaf630195, State: Initialized, Role: FOLLOWER
I20260927 02:49:19.013242 12139 consensus_queue.cc:260] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.013533 12139 ts_tablet_manager.cc:1428] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260927 02:49:19.013661 11846 heartbeater.cc:503] Master 127.0.98.62:38923 was elected leader, sending a full tablet report...
I20260927 02:49:19.015364 12141 tablet_bootstrap.cc:492] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49: No bootstrap required, opened a new log
I20260927 02:49:19.015457 12141 ts_tablet_manager.cc:1397] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260927 02:49:19.017678 12141 raft_consensus.cc:348] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.017880 12141 raft_consensus.cc:374] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:49:19.017946 12141 raft_consensus.cc:729] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e89ad078fc427c9f4d859af5091d49, State: Initialized, Role: FOLLOWER
I20260927 02:49:19.018090 12141 consensus_queue.cc:260] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.018355 12141 ts_tablet_manager.cc:1428] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:49:19.018513 12120 heartbeater.cc:503] Master 127.0.98.62:38923 was elected leader, sending a full tablet report...
Sep 27 02:49:19 dist-test-slave-f4rg krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:49:19.078989 12146 raft_consensus.cc:482] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:49:19.079124 12146 raft_consensus.cc:504] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.079548 12146 leader_election.cc:288] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f00bd10da5e4a17ba8c840cc87aedc9 (127.0.98.2:44759), 3238ec3768e944e6894b516eaf630195 (127.0.98.1:39507)
I20260927 02:49:19.083231 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:19.079664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44759 (local address 127.0.98.3:40555)
0927 02:49:19.079782 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:19.079793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:19.079838 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:19.080111 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:19.080112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:19.080114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:49:19.080183 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:19.080185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.081136 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.081137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:19.082255 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:19.082259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.082982 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.082983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:19.083059 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:19.083063 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:49:19.083102 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 02:49:19.083143 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 02:49:19.083593 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:19.079918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40555 (local address 127.0.98.2:44759)
0927 02:49:19.079965 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:19.079967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:19.079976 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:49:19.080003 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:19.080004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:19.080012 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:19.080059 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:49:19.080268 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.081017 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.082354 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.082874 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.082938 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:19.082951 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:19.083488 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 02:49:19.083533 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:49:19.083940 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c879802953b48bda0672b4a757b0991" candidate_uuid: "35e89ad078fc427c9f4d859af5091d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" is_pre_election: true
I20260927 02:49:19.084132 11939 raft_consensus.cc:2454] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e89ad078fc427c9f4d859af5091d49 in term 0.
I20260927 02:49:19.084368 12007 leader_election.cc:302] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e89ad078fc427c9f4d859af5091d49, 6f00bd10da5e4a17ba8c840cc87aedc9; no voters:
I20260927 02:49:19.084513 12146 raft_consensus.cc:2781] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:49:19.084570 12146 raft_consensus.cc:482] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:49:19.084592 12146 raft_consensus.cc:3037] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:49:19.085302 12146 raft_consensus.cc:504] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.085500 12146 leader_election.cc:288] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [CANDIDATE]: Term 1 election: Requested vote from peers 6f00bd10da5e4a17ba8c840cc87aedc9 (127.0.98.2:44759), 3238ec3768e944e6894b516eaf630195 (127.0.98.1:39507)
I20260927 02:49:19.085645 11939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c879802953b48bda0672b4a757b0991" candidate_uuid: "35e89ad078fc427c9f4d859af5091d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9"
I20260927 02:49:19.085724 11939 raft_consensus.cc:3037] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:49:19.086480 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:19.079715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39507 (local address 127.0.98.3:51623)
0927 02:49:19.083315 (+ 3600us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:19.083328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:49:19.083375 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:49:19.083542 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:49:19.083544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:49:19.083546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:49:19.083605 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:19.083607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.084513 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.084514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:19.085529 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:49:19.085532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.086265 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.086266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:49:19.086323 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:19.086327 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:49:19.086367 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:49:19.086412 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3587,"thread_start_us":96,"threads_started":1}
I20260927 02:49:19.086601 11939 raft_consensus.cc:2454] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e89ad078fc427c9f4d859af5091d49 in term 1.
I20260927 02:49:19.086742 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:19.079972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51623 (local address 127.0.98.1:39507)
0927 02:49:19.080011 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:19.080013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:19.083375 (+ 3362us) server_negotiation.cc:407] Connection header received
0927 02:49:19.083420 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:19.083421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:19.083435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:19.083495 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:49:19.083693 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.084428 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.085623 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.086157 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.086224 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:19.086236 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:19.086627 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 02:49:19.086681 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:49:19.086935 12007 leader_election.cc:302] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e89ad078fc427c9f4d859af5091d49, 6f00bd10da5e4a17ba8c840cc87aedc9; no voters:
I20260927 02:49:19.087016 12146 raft_consensus.cc:2781] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:49:19.087001 11800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c879802953b48bda0672b4a757b0991" candidate_uuid: "35e89ad078fc427c9f4d859af5091d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3238ec3768e944e6894b516eaf630195" is_pre_election: true
I20260927 02:49:19.087152 11800 raft_consensus.cc:2454] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35e89ad078fc427c9f4d859af5091d49 in term 0.
I20260927 02:49:19.087167 12146 raft_consensus.cc:686] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 1 LEADER]: Becoming Leader. State: Replica: 35e89ad078fc427c9f4d859af5091d49, State: Running, Role: LEADER
I20260927 02:49:19.087253 12146 consensus_queue.cc:237] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } }
I20260927 02:49:19.087275 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c879802953b48bda0672b4a757b0991" candidate_uuid: "35e89ad078fc427c9f4d859af5091d49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3238ec3768e944e6894b516eaf630195"
I20260927 02:49:19.087328 11799 raft_consensus.cc:3037] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:49:19.088181 11799 raft_consensus.cc:2454] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35e89ad078fc427c9f4d859af5091d49 in term 1.
I20260927 02:49:19.088068 11636 catalog_manager.cc:5495] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e89ad078fc427c9f4d859af5091d49 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "35e89ad078fc427c9f4d859af5091d49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35e89ad078fc427c9f4d859af5091d49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44195 } health_report { overall_health: HEALTHY } } }
I20260927 02:49:19.109863 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:19.106559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.0.98.3:44195)
0927 02:49:19.106604 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:49:19.106605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:49:19.106613 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:49:19.106654 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:49:19.106655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:49:19.106678 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:49:19.106740 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:49:19.106981 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.107651 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.108725 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:49:19.109015 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:49:19.109072 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:49:19.109272 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:49:19.109412 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:49:19.109487 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:49:19.109741 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 02:49:19.109768 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:49:19.112229 11799 raft_consensus.cc:1261] T 8c879802953b48bda0672b4a757b0991 P 3238ec3768e944e6894b516eaf630195 [term 1 FOLLOWER]: Refusing update from remote peer 35e89ad078fc427c9f4d859af5091d49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:49:19.112666 12148 consensus_queue.cc:1035] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3238ec3768e944e6894b516eaf630195" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:49:19.115002 11939 raft_consensus.cc:1261] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 1 FOLLOWER]: Refusing update from remote peer 35e89ad078fc427c9f4d859af5091d49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:49:19.115365 12148 consensus_queue.cc:1035] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f00bd10da5e4a17ba8c840cc87aedc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:49:19.119642 12151 mvcc.cc:205] Tried to move back new op lower bound from 7333795262920589312 to 7333795262822060032. Current Snapshot: MvccSnapshot[applied={T|T < 7333795262920589312}]
I20260927 02:49:19.119894 12154 mvcc.cc:205] Tried to move back new op lower bound from 7333795262920589312 to 7333795262822060032. Current Snapshot: MvccSnapshot[applied={T|T < 7333795262920589312}]
I20260927 02:49:19.122257 12152 mvcc.cc:205] Tried to move back new op lower bound from 7333795262920589312 to 7333795262822060032. Current Snapshot: MvccSnapshot[applied={T|T < 7333795262920589312}]
I20260927 02:49:19.124974 11636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:49:19.125064 11636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:49:19.125857 11636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 174338e435434853897e2d23afde8d45: Sending DeleteTablet for 3 replicas of tablet 8c879802953b48bda0672b4a757b0991
I20260927 02:49:19.126231 11918 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c879802953b48bda0672b4a757b0991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:49:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44560
I20260927 02:49:19.126497 11762 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c879802953b48bda0672b4a757b0991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:49:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59962
I20260927 02:49:19.126886 12054 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c879802953b48bda0672b4a757b0991 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:49:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57724
I20260927 02:49:19.127589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 11717
I20260927 02:49:19.129263 12160 tablet_replica.cc:330] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: stopping tablet replica
I20260927 02:49:19.129447 12160 raft_consensus.cc:2229] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:49:19.129638 12160 raft_consensus.cc:2258] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:49:19.130158 12160 ts_tablet_manager.cc:1905] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:49:19.132270 12160 ts_tablet_manager.cc:1918] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:49:19.132406 12160 log.cc:1196] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/8c879802953b48bda0672b4a757b0991
I20260927 02:49:19.133052 12160 ts_tablet_manager.cc:1939] T 8c879802953b48bda0672b4a757b0991 P 6f00bd10da5e4a17ba8c840cc87aedc9: Deleting consensus metadata
I20260927 02:49:19.133555 11620 catalog_manager.cc:4841] TS 6f00bd10da5e4a17ba8c840cc87aedc9 (127.0.98.2:44759): tablet 8c879802953b48bda0672b4a757b0991 (table test-table [id=744347527bf7490ebb0fa92c176fa4e9]) successfully deleted
W20260927 02:49:19.134042 11621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39507 (error 108)
I20260927 02:49:19.134243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 11854
I20260927 02:49:19.138517 12162 tablet_replica.cc:330] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49: stopping tablet replica
I20260927 02:49:19.138743 12162 raft_consensus.cc:2229] T 8c879802953b48bda0672b4a757b0991 P 35e89ad078fc427c9f4d859af5091d49 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:49:19.138954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 11990
I20260927 02:49:19.140256 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:19.140111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39507 (local address 127.0.0.1:59966)
0927 02:49:19.140163 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:19.140215 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:49:19.140672 11621 catalog_manager.cc:4584] TS 3238ec3768e944e6894b516eaf630195 (127.0.98.1:39507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c879802953b48bda0672b4a757b0991: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39507: connect: Connection refused (error 111)
I20260927 02:49:19.142956 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:49:19.142861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44195 (local address 127.0.0.1:57738)
0927 02:49:19.142904 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:49:19.142926 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:44195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:49:19.143584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 11604
I20260927 02:49:19.147845 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:49:19.482523 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:49:19.482620 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:49:19.816898 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:49:19.817 UTC [11190] LOG: received smart shutdown request
2026-09-27 02:49:19.820 UTC [11190] LOG: background worker "logical replication launcher" (PID 11199) exited with exit code 1
2026-09-27 02:49:19.820 UTC [11194] LOG: shutting down
2026-09-27 02:49:19.969 UTC [11190] LOG: database system is shut down
2026-09-27T02:49:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44076 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:49:20 dist-test-slave-f4rg krb5kdc[12172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:49:20 dist-test-slave-f4rg krb5kdc[12172](info): set up 2 sockets
Sep 27 02:49:20 dist-test-slave-f4rg krb5kdc[12172](info): commencing operation
krb5kdc: starting...
W20260927 02:49:21.010494 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:49:21 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:49:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 02:49:21.157228 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskc9ab2Y/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/postgres -l logfile start
2026-09-27 02:49:23.423 UTC [12202] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 02:49:23.423 UTC [12202] LOG: listening on IPv4 address "127.0.98.20", port 35779
2026-09-27 02:49:23.426 UTC [12202] LOG: listening on Unix socket "/tmp/.s.PGSQL.35779"
2026-09-27 02:49:23.445 UTC [12205] LOG: database system was shut down at 2026-09-27 02:49:22 UTC
2026-09-27 02:49:23.448 UTC [12202] LOG: database system is ready to accept connections
I20260927 02:49:24.334470 392 mini_postgres.cc:96] Postgres bound to 35779
2026-09-27 02:49:24.340 UTC [12213] FATAL: database "slave" does not exist
127.0.98.20:35779 - accepting connections
I20260927 02:49:24.340480 392 mini_ranger.cc:161] Starting Ranger...
I20260927 02:49:24.354022 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 02:49:24.774580 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 02:49:24.774662 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-admin
2026-09-27 02:49:25,033 [I] DB FLAVOR :POSTGRES
2026-09-27 02:49:25,033 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:49:25,033 [I] Checking connection..
2026-09-27 02:49:25,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 02:49:25,288 [I] Checking connection passed.
2026-09-27 02:49:25,288 [I] --------- Verifying version history table ---------
2026-09-27 02:49:25,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:49:25,545 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:49:25,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:49:25,790 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 02:49:25,790 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 02:49:25,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 02:49:26.045 UTC [12277] WARNING: there is no transaction in progress
2026-09-27 02:49:26,050 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 02:49:26,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 02:49:26,292 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 02:49:26,293 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:49:26,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 02:49:26,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 02:49:26,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f4rg.c.gcp-upstream.internal','N') ;"
2026-09-27 02:49:27,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:49:27,270 [I] Table x_portal_user does not exist in database ranger
2026-09-27 02:49:27,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:49:27,503 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 02:49:27,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:49:27,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:49:27,983 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 02:49:27,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 02:49:28.917 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:28.932 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:28.948 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:28.963 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:28.978 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:28.995 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.048 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.054 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.068 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.078 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.088 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.101 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.110 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.116 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.122 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29.128 UTC [12385] WARNING: there is no transaction in progress
2026-09-27 02:49:29,747 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 02:49:29,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 02:49:29,988 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 02:49:29,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 02:49:30,221 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 02:49:30,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 02:49:30,474 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 02:49:30,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 02:49:30,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 02:49:30,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f4rg.c.gcp-upstream.internal';"
2026-09-27 02:49:31,185 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 02:49:31,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 02:49:31,419 [I] DB_PATCHES have already been applied
I20260927 02:49:31.425010 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskc9ab2Y/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 02:49:31.425074 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 02:49:31.427842 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59353
Sep 27, 2026 2:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 2:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 2:49:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 2:49:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 2:49:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 02:49:32.415632 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.075s user 0.000s sys 0.008s
Sep 27, 2026 2:49:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 2:49:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 2:49:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 2:49:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 2:49:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 02:49:52.554710 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 02:49:52 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477392, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 02:49:52.634050 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 02:49:52.642561 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 02:49:53.421054 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 02:49:53.421126 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms
I20260927 02:49:53.421142 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35779
2026-09-27 02:49:53,564 [I] DB FLAVOR :POSTGRES
2026-09-27 02:49:53,564 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 02:49:53,565 [I] Checking connection
2026-09-27 02:49:53,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 02:49:53,788 [I] connection success
2026-09-27 02:49:53,789 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 02:49:53,789 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 02:49:53,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 02:49:54,025 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 02:49:54,025 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 02:49:54,025 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 02:49:54,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35779/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 02:49:54,278 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 02:49:54.714745 392 mini_ranger_kms.cc:310] Created kms service
I20260927 02:49:54.935065 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 02:49:54.976028 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 02:49:54 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477392, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:49:55.325739 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 02:49:55.325937 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskc9ab2Y/build/release/bin/postgresql.jar:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskc9ab2Y/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 02:49:55.325959 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 02:49:55.328986 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12565
I20260927 02:49:55.329062 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48619
Sep 27, 2026 2:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 2:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 2:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 2:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 2:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 2:49:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 02:49:57.349510 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.715s user 0.010s sys 0.116s
I20260927 02:49:57.349617 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 02:49:57.349668 392 mini_ranger_kms.cc:401] 127.0.98.20:48619/kms/v1/keys
Sep 27 02:49:57 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477397, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 02:49:58 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477397, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:49:58 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477397, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 02:50:00 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:50:00.673388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36633
--ranger_config_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:00.785807 12616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:00.786083 12616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:00.786116 12616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:00.787837 12616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:00.787899 12616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:00.787919 12616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:00.787936 12616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:00.787956 12616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:00.790186 12616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48619/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36633
--ranger_config_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:00.790441 12616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:00.790709 12616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:00.795665 12622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:00.795729 12621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:00.795873 12624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:00.795894 12616 server_base.cc:1034] running on GCE node
I20260927 02:50:00.796160 12616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:00.796500 12616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:00.797681 12616 hybrid_clock.cc:648] HybridClock initialized: now 1790477400797653 us; error 41 us; skew 500 ppm
Sep 27 02:50:00 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:00.800383 12616 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:00.800805 12616 webserver.cc:460] Webserver started at http://127.0.98.62:40213/ using document root <none> and password file <none>
I20260927 02:50:00.801002 12616 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:00.801041 12616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:00.801163 12616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:50:00 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:00.845569 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "ccd551f5477246339d5932e0294d4aae"
format_stamp: "Formatted at 2026-09-27 02:50:00 on dist-test-slave-f4rg"
server_key: "c4661064f7c10660e92b3e79c3db6d0d"
server_key_iv: "2679b9b9b28aa30deef7dc60206f835e"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:00.846058 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccd551f5477246339d5932e0294d4aae"
format_stamp: "Formatted at 2026-09-27 02:50:00 on dist-test-slave-f4rg"
server_key: "c4661064f7c10660e92b3e79c3db6d0d"
server_key_iv: "2679b9b9b28aa30deef7dc60206f835e"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:00.847903 12616 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 02:50:00.860415 12631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:00.860584 12616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:00.860679 12616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "ccd551f5477246339d5932e0294d4aae"
format_stamp: "Formatted at 2026-09-27 02:50:00 on dist-test-slave-f4rg"
server_key: "c4661064f7c10660e92b3e79c3db6d0d"
server_key_iv: "2679b9b9b28aa30deef7dc60206f835e"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:00.860791 12616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:00.871897 12616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:00.873827 12616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:00.873950 12616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:00.878226 12616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36633
I20260927 02:50:00.878295 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36633 every 8 connection(s)
I20260927 02:50:00.878612 12616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:00.879012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 12616
I20260927 02:50:00.879122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:50:00.879299 392 external_mini_cluster.cc:1428] Setting key ee4c3a4eddeb2c4ac3011453e9f14727
I20260927 02:50:00.879302 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:00.882259 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Bootstrap starting.
I20260927 02:50:00.882910 12684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:00.883154 12684 log.cc:824] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:00.883888 12684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: No bootstrap required, opened a new log
I20260927 02:50:00.886049 12684 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:00.886225 12684 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:00.886257 12684 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd551f5477246339d5932e0294d4aae, State: Initialized, Role: FOLLOWER
I20260927 02:50:00.886421 12684 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:00.886511 12684 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:00.886543 12684 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:00.886611 12684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:50:00 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:00.887214 12684 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:00.887322 12684 leader_election.cc:302] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd551f5477246339d5932e0294d4aae; no voters:
I20260927 02:50:00.887485 12684 leader_election.cc:288] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:50:00.887763 12689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:00.887835 12684 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:00.888240 12689 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 LEADER]: Becoming Leader. State: Replica: ccd551f5477246339d5932e0294d4aae, State: Running, Role: LEADER
I20260927 02:50:00.888399 12689 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:00.889029 12689 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd551f5477246339d5932e0294d4aae. Latest consensus state: current_term: 1 leader_uuid: "ccd551f5477246339d5932e0294d4aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260927 02:50:00.889184 12689 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:00.889257 12684 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:50:00.889433 12703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:00.892575 12703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:00.894436 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:00.879791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.0.98.62:36633)
0927 02:50:00.879979 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:00.879988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:00.880018 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:50:00.880723 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:00.880733 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:00.880799 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:00.880952 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:00.881658 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:00.883180 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:00.884118 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:00.884746 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:00.884851 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:00.887595 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:00.887599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:00.887600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:00.887613 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:00.889581 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:00.891032 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:00.891083 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:00.891083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:00.891524 (+ 441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:00.891782 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:00.891782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:00.891783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:00.892622 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:00.893377 (+ 755us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:00.893469 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:50:00.894095 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260927 02:50:00.894486 12690 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccd551f5477246339d5932e0294d4aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260927 02:50:00.895900 12690 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:00.896909 12703 catalog_manager.cc:1347] Generated new cluster ID: 80afe2aeb1624b11af07c9db540868db
I20260927 02:50:00.896955 12703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:50:00.922847 12703 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:50:00.923399 12703 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:00.935920 12703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Generated new TSK 0
I20260927 02:50:00.936161 12703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:50:01.589368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:01.706954 12729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:01.707249 12729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:01.707286 12729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:01.708976 12729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:01.709039 12729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:01.709079 12729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:50:01.711241 12729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:01.711485 12729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:01.711790 12729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:01.718691 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:01.718686 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:01.718925 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:01.719630 12729 server_base.cc:1034] running on GCE node
I20260927 02:50:01.719832 12729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:01.720081 12729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:01.721289 12729 hybrid_clock.cc:648] HybridClock initialized: now 1790477401721271 us; error 32 us; skew 500 ppm
Sep 27 02:50:01 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:01.723824 12729 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:50:01.724236 12729 webserver.cc:460] Webserver started at http://127.0.98.1:35189/ using document root <none> and password file <none>
I20260927 02:50:01.724426 12729 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:01.724472 12729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:01.724604 12729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:50:01 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:01.738463 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "914369d36cf8481f9235fbb664b7cc73"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "08ee5bf6a71dcfcdcac9c04d4a2e84c9"
server_key_iv: "e854133258d7caf8db10257487f2f33a"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:01.739148 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "914369d36cf8481f9235fbb664b7cc73"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "08ee5bf6a71dcfcdcac9c04d4a2e84c9"
server_key_iv: "e854133258d7caf8db10257487f2f33a"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:01.740940 12729 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 02:50:01.754756 12745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:01.755013 12729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:01.755118 12729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "914369d36cf8481f9235fbb664b7cc73"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "08ee5bf6a71dcfcdcac9c04d4a2e84c9"
server_key_iv: "e854133258d7caf8db10257487f2f33a"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:01.755213 12729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:01.790169 12729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:01.792523 12729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:01.792706 12729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:01.792996 12729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:01.793449 12729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:01.793502 12729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:01.793532 12729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:01.793629 12729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:01.800616 12729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39401
I20260927 02:50:01.800678 12859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39401 every 8 connection(s)
I20260927 02:50:01.801168 12729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:50:01.807366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 12729
I20260927 02:50:01.807453 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:50:01.807646 392 external_mini_cluster.cc:1428] Setting key 22c471dc8d37e5e7e0e3ea676004aee3
Sep 27 02:50:01 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:01.812047 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:01.801928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37509 (local address 127.0.98.62:36633)
0927 02:50:01.802057 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:01.802059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:01.802153 (+ 94us) server_negotiation.cc:407] Connection header received
0927 02:50:01.802839 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:01.802840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:01.802854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:01.802932 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:01.803609 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:01.804408 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:01.805850 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:01.806051 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:01.806104 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:01.808533 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:01.808534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:01.808534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:01.808542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:01.810546 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:01.810969 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:01.810970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:01.810971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:01.811038 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:01.811315 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:01.811316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:01.811316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:01.811448 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:01.811510 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:01.811860 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 02:50:01.811948 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260927 02:50:01.812233 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:01.801851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.1:37509)
0927 02:50:01.802009 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:01.802059 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:01.802702 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:01.803010 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:01.803015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:01.803249 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:01.803518 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:01.803524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:01.804512 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:01.804513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:01.805761 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:01.805764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:01.806152 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:01.806153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:01.806291 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:01.806784 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:01.806795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:01.808424 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:01.810636 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:01.810643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:01.810645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:01.810909 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:01.811093 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:01.811094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:01.811094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:01.811243 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:01.811518 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:01.811524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:01.811716 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 02:50:01.811833 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:50:01.812000 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260927 02:50:01.812721 12860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260927 02:50:01.812815 12860 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:01.812994 12860 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:01.813565 12648 ts_manager.cc:194] Registered new tserver with Master: 914369d36cf8481f9235fbb664b7cc73 (127.0.98.1:39401)
I20260927 02:50:01.814426 12648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37509
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:50:01.833705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:50:01 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 02:50:01.946326 12866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:01.946516 12866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:01.946545 12866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:01.948222 12866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:01.948333 12866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:01.948418 12866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:50:01.950551 12866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:01.951004 12866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:01.951350 12866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:50:01.957275 12866 server_base.cc:1034] running on GCE node
W20260927 02:50:01.959273 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:01.959506 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:01.960332 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:01.960616 12866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:01.960891 12866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:01.962054 12866 hybrid_clock.cc:648] HybridClock initialized: now 1790477401962034 us; error 33 us; skew 500 ppm
Sep 27 02:50:01 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:01.965176 12866 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:50:01.965677 12866 webserver.cc:460] Webserver started at http://127.0.98.2:37383/ using document root <none> and password file <none>
I20260927 02:50:01.965905 12866 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:01.965952 12866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:01.966069 12866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:50:01 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:01.982236 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "084bbf0940a04d8182bb5130407f541c"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "454dca212ffc4ea7577d29a141cc6f0b"
server_key_iv: "3d69715d3563ea9a2766ce0ef3838dcd"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:01.982663 12866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "084bbf0940a04d8182bb5130407f541c"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "454dca212ffc4ea7577d29a141cc6f0b"
server_key_iv: "3d69715d3563ea9a2766ce0ef3838dcd"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:01.984382 12866 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:50:01.998569 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:01.998766 12866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:01.998886 12866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "084bbf0940a04d8182bb5130407f541c"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "454dca212ffc4ea7577d29a141cc6f0b"
server_key_iv: "3d69715d3563ea9a2766ce0ef3838dcd"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:01.998980 12866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:02.023464 12866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:02.025923 12866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:02.026125 12866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:02.026423 12866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:02.026875 12866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:02.026959 12866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.027024 12866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:02.027079 12866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.034390 12866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46133
I20260927 02:50:02.034925 12866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:50:02.035588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 12866
I20260927 02:50:02.035687 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:50:02.035871 392 external_mini_cluster.cc:1428] Setting key 6f67e00b05d6648d7d57038b6be64521
I20260927 02:50:02.043533 12997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46133 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:50:02.063978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:02.065852 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.044577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58313 (local address 127.0.98.62:36633)
0927 02:50:02.044619 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:02.044621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:02.053500 (+ 8879us) server_negotiation.cc:407] Connection header received
0927 02:50:02.054271 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:02.054273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:02.054289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:02.054319 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:02.055210 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.055973 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.057116 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.057394 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.057457 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.060268 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:02.060269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:02.060270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:02.060278 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:02.062596 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.063046 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.063047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.063048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.063107 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.063437 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.063438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.063438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.063571 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:02.063621 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:02.065630 (+ 2009us) server_negotiation.cc:299] Negotiation successful
0927 02:50:02.065736 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:50:02.066573 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.044265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.2:58313)
0927 02:50:02.053307 (+ 9042us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:02.053376 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:02.054120 (+ 744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:02.054522 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:02.054527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:02.054762 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:02.055084 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.055091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.056094 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.056095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.057004 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.057007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.057601 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.057602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.057755 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.058350 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:02.058360 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:02.060093 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:02.062681 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.062688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.062691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.062974 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.063174 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.063175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.063175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.063363 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.065255 (+ 1892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:02.065262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:02.065463 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 02:50:02.066172 (+ 709us) client_negotiation.cc:240] Negotiation successful
0927 02:50:02.066340 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8976,"thread_start_us":94,"threads_started":1}
I20260927 02:50:02.067332 12998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260927 02:50:02.067459 12998 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.067705 12998 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:02.068102 12648 ts_manager.cc:194] Registered new tserver with Master: 084bbf0940a04d8182bb5130407f541c (127.0.98.2:46133)
I20260927 02:50:02.068567 12648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58313
W20260927 02:50:02.174264 13003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:02.174561 13003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:02.174604 13003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:02.176355 13003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:02.176419 13003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:02.176463 13003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:50:02.178756 13003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:02.179039 13003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:02.179335 13003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:02.185850 13011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:02.187037 13003 server_base.cc:1034] running on GCE node
W20260927 02:50:02.187228 13009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.187383 13008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:02.187716 13003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:02.188017 13003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:02.189181 13003 hybrid_clock.cc:648] HybridClock initialized: now 1790477402189147 us; error 47 us; skew 500 ppm
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:02.192631 13003 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:50:02.193162 13003 webserver.cc:460] Webserver started at http://127.0.98.3:41667/ using document root <none> and password file <none>
I20260927 02:50:02.193421 13003 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:02.193477 13003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:02.193621 13003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:02.210175 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "8baeb34efeea467f9e0f6be0672065e2"
format_stamp: "Formatted at 2026-09-27 02:50:02 on dist-test-slave-f4rg"
server_key: "78fbc7e5a935ae5b0bd15aa82c2d7c16"
server_key_iv: "9cec17aa1188452f0457dac820360841"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:02.210595 13003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8baeb34efeea467f9e0f6be0672065e2"
format_stamp: "Formatted at 2026-09-27 02:50:02 on dist-test-slave-f4rg"
server_key: "78fbc7e5a935ae5b0bd15aa82c2d7c16"
server_key_iv: "9cec17aa1188452f0457dac820360841"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:02.212354 13003 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 02:50:02.224797 13018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.225026 13003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:02.225104 13003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "8baeb34efeea467f9e0f6be0672065e2"
format_stamp: "Formatted at 2026-09-27 02:50:02 on dist-test-slave-f4rg"
server_key: "78fbc7e5a935ae5b0bd15aa82c2d7c16"
server_key_iv: "9cec17aa1188452f0457dac820360841"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:02.225184 13003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:02.253579 13003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:02.256089 13003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:02.256237 13003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:02.256450 13003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:02.256852 13003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:02.256922 13003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.256975 13003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:02.257012 13003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.264353 13003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42181
I20260927 02:50:02.264907 13003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:50:02.265420 13131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42181 every 8 connection(s)
I20260927 02:50:02.271662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13003
I20260927 02:50:02.271767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:02.271931 392 external_mini_cluster.cc:1428] Setting key 52d1edcf831f847121fb70820607563c
I20260927 02:50:02.275360 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.265973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36181 (local address 127.0.98.62:36633)
0927 02:50:02.266013 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:02.266015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:02.266025 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:50:02.266582 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:02.266583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:02.266596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:02.266654 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:02.267371 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.268113 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.269517 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.269794 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.269863 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.272308 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:02.272309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:02.272310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:02.272318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:02.273868 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.274316 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.274317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.274317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.274380 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.274629 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.274630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.274631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.274767 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:02.274864 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:02.275191 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 02:50:02.275269 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:50:02.275825 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.265691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.3:36181)
0927 02:50:02.265851 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:02.265894 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:02.266473 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:02.266736 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:02.266741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:02.267004 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:02.267275 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.267282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.268213 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.268214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.269422 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.269425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.269888 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.269889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.270022 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.270588 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:02.270598 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:02.272198 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:02.273952 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.273959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.273962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.274245 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.274442 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.274442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.274443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.274570 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.274868 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:02.274874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:02.275052 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 02:50:02.275162 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 02:50:02.275309 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20260927 02:50:02.276473 13132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260927 02:50:02.276612 13132 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.276846 13132 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:02.277275 12648 ts_manager.cc:194] Registered new tserver with Master: 8baeb34efeea467f9e0f6be0672065e2 (127.0.98.3:42181)
I20260927 02:50:02.277752 12648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36181
I20260927 02:50:02.283653 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:50:02.284303 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 12729
I20260927 02:50:02.289860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 12866
I20260927 02:50:02.299801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13003
I20260927 02:50:02.304417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 12616
I20260927 02:50:02.308946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--webserver_interface=127.0.98.62
--webserver_port=40213
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36633
--ranger_config_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:02.389988 13135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:02.390197 13135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:02.390218 13135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:02.391394 13135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:02.391438 13135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:02.391451 13135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:02.391462 13135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:02.391474 13135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:02.393050 13135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36633
--ranger_config_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:02.393265 13135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:02.393484 13135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:02.398100 13141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:02.398236 13135 server_base.cc:1034] running on GCE node
W20260927 02:50:02.398329 13140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.398209 13143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:02.398577 13135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:02.398881 13135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:02.400029 13135 hybrid_clock.cc:648] HybridClock initialized: now 1790477402400014 us; error 31 us; skew 500 ppm
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:02.402310 13135 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:02.402745 13135 webserver.cc:460] Webserver started at http://127.0.98.62:40213/ using document root <none> and password file <none>
I20260927 02:50:02.402978 13135 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:02.403029 13135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:02.448527 13135 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:02.449189 13151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.449354 13135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:02.449440 13135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "ccd551f5477246339d5932e0294d4aae"
format_stamp: "Formatted at 2026-09-27 02:50:00 on dist-test-slave-f4rg"
server_key: "c4661064f7c10660e92b3e79c3db6d0d"
server_key_iv: "2679b9b9b28aa30deef7dc60206f835e"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:02.449797 13135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:02.461671 13135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:02.463537 13135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:02.463665 13135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:02.467957 13135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36633
I20260927 02:50:02.468021 13203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36633 every 8 connection(s)
I20260927 02:50:02.468372 13135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:02.469924 13204 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:50:02.470794 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Bootstrap starting.
I20260927 02:50:02.474773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13135
I20260927 02:50:02.475049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39401
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36633
--webserver_port=35189
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:02.480651 13204 log.cc:824] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:02.482887 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:50:02.483208 13204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Bootstrap complete.
I20260927 02:50:02.486426 13204 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:02.486780 13204 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd551f5477246339d5932e0294d4aae, State: Initialized, Role: FOLLOWER
I20260927 02:50:02.486985 13204 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:02.487061 13204 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:02.487102 13204 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:02.487150 13204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:50:02.488462 13204 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:02.488567 13204 leader_election.cc:302] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd551f5477246339d5932e0294d4aae; no voters:
I20260927 02:50:02.488736 13204 leader_election.cc:288] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:50:02.488857 13208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:50:02.489018 13208 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [term 2 LEADER]: Becoming Leader. State: Replica: ccd551f5477246339d5932e0294d4aae, State: Running, Role: LEADER
I20260927 02:50:02.489061 13204 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:02.489185 13208 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260927 02:50:02.489475 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccd551f5477246339d5932e0294d4aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260927 02:50:02.489585 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:02.489753 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd551f5477246339d5932e0294d4aae. Latest consensus state: current_term: 2 leader_uuid: "ccd551f5477246339d5932e0294d4aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd551f5477246339d5932e0294d4aae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260927 02:50:02.489837 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:02.489959 13217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:02.490432 13217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:02.490586 13217 catalog_manager.cc:1259] Loaded cluster ID: 80afe2aeb1624b11af07c9db540868db
I20260927 02:50:02.490661 13217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:50:02.490768 13204 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 02:50:02.494014 13217 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:02.494479 13217 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Loaded TSK: 0
I20260927 02:50:02.494737 13217 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:50:02.583904 13206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:02.584096 13206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:02.584134 13206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:02.585278 13206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:02.585317 13206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:02.585352 13206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:50:02.587152 13206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39401
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35189
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:02.587375 13206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:02.587603 13206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:02.592504 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.592671 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.592706 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:02.593314 13206 server_base.cc:1034] running on GCE node
I20260927 02:50:02.593492 13206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:02.593715 13206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:02.594847 13206 hybrid_clock.cc:648] HybridClock initialized: now 1790477402594824 us; error 32 us; skew 500 ppm
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:02.600354 13206 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:50:02.600813 13206 webserver.cc:460] Webserver started at http://127.0.98.1:35189/ using document root <none> and password file <none>
I20260927 02:50:02.601032 13206 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:02.601094 13206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:02.617208 13206 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:50:02.618999 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.619179 13206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 02:50:02.619252 13206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "914369d36cf8481f9235fbb664b7cc73"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "08ee5bf6a71dcfcdcac9c04d4a2e84c9"
server_key_iv: "e854133258d7caf8db10257487f2f33a"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:02.619606 13206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:02.648751 13206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:02.651090 13206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:02.651257 13206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:02.651510 13206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:02.651866 13206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:02.651909 13206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.651940 13206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:02.651957 13206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.658025 13206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39401
I20260927 02:50:02.658397 13206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:50:02.665227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13206
I20260927 02:50:02.665506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46133
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36633
--webserver_port=37383
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:02.670233 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.659204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43885 (local address 127.0.98.62:36633)
0927 02:50:02.659420 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:02.659429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:02.659788 (+ 359us) server_negotiation.cc:407] Connection header received
0927 02:50:02.660469 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:02.660474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:02.660519 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:02.660648 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:02.661456 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.662924 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.663974 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.664390 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.664482 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.666977 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:02.666982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:02.666982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:02.666994 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:02.668504 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.668879 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.668883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.668884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.668988 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.669271 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.669272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.669272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.669541 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:02.669590 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:02.669927 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 02:50:02.670046 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260927 02:50:02.670739 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.659465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.1:43885)
0927 02:50:02.659623 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:02.659663 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:02.660235 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:02.660802 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:02.660806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:02.661055 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:02.661322 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.661329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.663050 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.663051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.663891 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.663893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.664470 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.664471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.664592 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.665096 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:02.665107 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:02.666841 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:02.668580 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.668587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.668589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.668806 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.669062 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.669063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.669063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.669191 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.669628 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:02.669633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:02.669750 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 02:50:02.670399 (+ 649us) client_negotiation.cc:240] Negotiation successful
0927 02:50:02.670549 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260927 02:50:02.671409 13365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39401 every 8 connection(s)
I20260927 02:50:02.671447 13366 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260927 02:50:02.671545 13366 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.671717 13366 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
W20260927 02:50:02.672478 13366 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36633 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:50:02.672540 13366 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.672601 13366 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:02.672878 13366 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.672950 13366 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
W20260927 02:50:02.673223 13366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:50:02.774158 13370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:02.774354 13370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:02.774384 13370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:02.775535 13370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:02.775575 13370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:02.775609 13370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:50:02.777133 13370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37383
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:02.777359 13370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:02.777613 13370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:02.783234 13375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.783510 13376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.784358 13378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:02.784904 13370 server_base.cc:1034] running on GCE node
I20260927 02:50:02.785120 13370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:02.785380 13370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:02.786563 13370 hybrid_clock.cc:648] HybridClock initialized: now 1790477402786550 us; error 30 us; skew 500 ppm
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:02.789606 13370 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:50:02.790057 13370 webserver.cc:460] Webserver started at http://127.0.98.2:37383/ using document root <none> and password file <none>
I20260927 02:50:02.790277 13370 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:02.790331 13370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:02.815783 13370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260927 02:50:02.816737 13386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.816931 13370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:02.817018 13370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "084bbf0940a04d8182bb5130407f541c"
format_stamp: "Formatted at 2026-09-27 02:50:01 on dist-test-slave-f4rg"
server_key: "454dca212ffc4ea7577d29a141cc6f0b"
server_key_iv: "3d69715d3563ea9a2766ce0ef3838dcd"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:02.817471 13370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:02.835611 13370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:02.837921 13370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:02.838079 13370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:02.838335 13370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:02.838755 13370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:02.838861 13370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.838907 13370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:02.838924 13370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.845927 13370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46133
I20260927 02:50:02.845991 13499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46133 every 8 connection(s)
I20260927 02:50:02.846406 13370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:50:02.849087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13370
I20260927 02:50:02.849352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48619/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42181
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36633
--webserver_port=41667
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:02.857236 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.847135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43319 (local address 127.0.98.62:36633)
0927 02:50:02.847206 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:02.847209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:02.847418 (+ 209us) server_negotiation.cc:407] Connection header received
0927 02:50:02.848113 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:02.848114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:02.848124 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:02.848167 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:02.848908 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.849634 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.851200 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.851492 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.851558 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.854058 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:02.854059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:02.854060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:02.854068 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:02.855716 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.856179 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.856180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.856180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.856239 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:02.856520 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:02.856520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:02.856521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:02.856651 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:02.856722 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:02.857058 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 02:50:02.857140 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:50:02.857424 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:02.847112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.2:43319)
0927 02:50:02.847273 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:02.847324 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:02.848013 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:02.848247 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:02.848252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:02.848489 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:02.848788 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.848797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.849727 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.849728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.851091 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:02.851094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:02.851597 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:02.851598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:02.851781 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:02.852302 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:02.852312 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:02.853958 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:02.855803 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.855810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.855812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.856100 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.856309 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:02.856310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:02.856311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:02.856451 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:02.856736 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:02.856742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:02.856913 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 02:50:02.857034 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 02:50:02.857194 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260927 02:50:02.858032 13500 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260927 02:50:02.858170 13500 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.858394 13500 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
W20260927 02:50:02.861799 13500 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36633 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:50:02.861931 13500 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.862028 13500 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:02.862325 13500 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:02.862421 13500 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
W20260927 02:50:02.862715 13500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:50:02.944767 13503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:02.944965 13503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:02.944995 13503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:02.946615 13503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:02.946682 13503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:02.946722 13503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:50:02.948968 13503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:48619/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41667
--tserver_master_addrs=127.0.98.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:02.949232 13503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:02.949482 13503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:02.955940 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.955901 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:02.956369 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:02.956396 13503 server_base.cc:1034] running on GCE node
I20260927 02:50:02.956594 13503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:02.956868 13503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:02.958034 13503 hybrid_clock.cc:648] HybridClock initialized: now 1790477402958014 us; error 34 us; skew 500 ppm
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:02.961216 13503 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:50:02.961761 13503 webserver.cc:460] Webserver started at http://127.0.98.3:41667/ using document root <none> and password file <none>
I20260927 02:50:02.962011 13503 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:02.962097 13503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 02:50:02 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:02.981745 13503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:02.982529 13520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:02.982730 13503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:50:02.982837 13503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "8baeb34efeea467f9e0f6be0672065e2"
format_stamp: "Formatted at 2026-09-27 02:50:02 on dist-test-slave-f4rg"
server_key: "78fbc7e5a935ae5b0bd15aa82c2d7c16"
server_key_iv: "9cec17aa1188452f0457dac820360841"
server_key_version: "kuduclusterkey@0"
I20260927 02:50:02.983265 13503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:03.014766 13503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:03.016944 13503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:03.017074 13503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:03.017318 13503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:03.017670 13503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:03.017714 13503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:03.017745 13503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:03.017765 13503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:03.024148 13503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42181
I20260927 02:50:03.024554 13503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:50:03.027992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13503
I20260927 02:50:03.025175 13633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42181 every 8 connection(s)
Sep 27 02:50:03 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:03.036193 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:03.025587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43647 (local address 127.0.98.62:36633)
0927 02:50:03.025637 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:03.025639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:03.025854 (+ 215us) server_negotiation.cc:407] Connection header received
0927 02:50:03.026350 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:03.026351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:03.026364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:03.026412 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:03.027008 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:03.027513 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:03.028898 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:03.029178 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:03.029232 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:03.032917 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:03.032918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:03.032919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:03.032925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:03.034387 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:03.035099 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:03.035101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:03.035101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:03.035167 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:03.035456 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:03.035457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:03.035457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:03.035581 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:03.035627 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:03.035999 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 02:50:03.036080 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:50:03.036726 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:03.025591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36633 (local address 127.0.98.3:43647)
0927 02:50:03.025739 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:03.025780 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:03.026252 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:03.026485 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:03.026490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:03.026683 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:03.026926 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:03.026931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:03.027601 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:03.027602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:03.028817 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:03.028820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:03.029307 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:03.029308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:03.029444 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:03.030001 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:03.030011 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:03.032809 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:03.034461 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:03.034468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:03.034471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:03.035012 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:03.035240 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:03.035241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:03.035241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:03.035381 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:03.035657 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:03.035664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:03.035853 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 02:50:03.036332 (+ 479us) client_negotiation.cc:240] Negotiation successful
0927 02:50:03.036494 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260927 02:50:03.037197 13634 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36633
I20260927 02:50:03.037288 13634 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:03.037483 13634 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:03.037607 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:03.028389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.0.98.62:36633)
0927 02:50:03.029089 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:03.029091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:03.029096 (+ 5us) server_negotiation.cc:407] Connection header received
0927 02:50:03.029112 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:03.029113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:03.029122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:03.029178 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:03.029938 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:03.030876 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:03.031859 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:03.032064 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:03.032115 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:03.033438 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:03.033439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:03.033440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:03.033446 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:03.035563 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:03.035916 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:03.035917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:03.035917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:03.035969 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:03.036157 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:03.036158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:03.036158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:03.036510 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:03.036674 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:03.037404 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 02:50:03.037480 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":64,"threads_started":1}
W20260927 02:50:03.038064 13634 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36633 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 02:50:03.038132 13634 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:03.038206 13634 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:03.038448 13634 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:03.038514 13634 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
W20260927 02:50:03.038795 13634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:50:03 dist-test-slave-f4rg krb5kdc[12172](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790477403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:03.673416 13366 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:03.673671 13366 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:03.674399 13168 ts_manager.cc:194] Registered new tserver with Master: 914369d36cf8481f9235fbb664b7cc73 (127.0.98.1:39401)
I20260927 02:50:03.675621 13168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43885
I20260927 02:50:03.862984 13500 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:03.863143 13500 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:03.865072 13167 ts_manager.cc:194] Registered new tserver with Master: 084bbf0940a04d8182bb5130407f541c (127.0.98.2:46133)
I20260927 02:50:03.865650 13167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43319
Sep 27 02:50:03 dist-test-slave-f4rg krb5kdc[12172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 02:50:04.038977 13634 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:04.039132 13634 heartbeater.cc:511] Master 127.0.98.62:36633 requested a full tablet report, sending...
I20260927 02:50:04.039439 13167 ts_manager.cc:194] Registered new tserver with Master: 8baeb34efeea467f9e0f6be0672065e2 (127.0.98.3:42181)
I20260927 02:50:04.039855 13167 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43647
I20260927 02:50:04.046066 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:50:04.054877 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.047072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45110 (local address 127.0.98.62:36633)
0927 02:50:04.047189 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:04.047191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:04.047251 (+ 60us) server_negotiation.cc:407] Connection header received
0927 02:50:04.047317 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:04.047318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:04.047329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:04.047394 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:04.048147 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.048908 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.050014 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.050309 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.050369 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.051839 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:04.051841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:04.051841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:04.051849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:04.053456 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:04.053831 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:04.053832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:04.053833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:04.053899 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:04.054055 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:04.054056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:04.054056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:04.054178 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:04.054259 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:04.054653 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 02:50:04.054743 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260927 02:50:04.057370 13167 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:50:04.059597 13167 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:50:04.068022 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.062945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42181 (local address 127.0.0.1:48294)
0927 02:50:04.063276 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:04.063286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:04.063331 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:04.063850 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:04.063854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:04.063857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:04.063932 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.063934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.065324 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.065325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.066584 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.066588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.067682 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.067683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.067795 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.067811 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:50:04.067880 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:50:04.067933 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":61,"threads_started":1}
I20260927 02:50:04.068027 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.063025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.0.98.3:42181)
0927 02:50:04.063674 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:04.063678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:04.063687 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:50:04.063724 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:04.063726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:04.063741 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:04.063823 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:04.064017 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.065240 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.066678 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.067591 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.067664 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.067747 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:04.067903 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:50:04.067948 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":59,"threads_started":1}
I20260927 02:50:04.069128 13568 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc6748322f6429c92079f279573c8c7 (DEFAULT_TABLE table=test-table [id=48a7643258f8425798444acfd0996bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:04.069475 13568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc6748322f6429c92079f279573c8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:04.074786 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.063150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46133 (local address 127.0.0.1:37032)
0927 02:50:04.063518 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:04.063528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:04.063570 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:04.071108 (+ 7538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:04.071109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:04.071112 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:04.071181 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.071184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.072520 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.072521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.073677 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.073679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.074559 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.074560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.074620 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.074626 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:50:04.074652 (+ 26us) client_negotiation.cc:240] Negotiation successful
0927 02:50:04.074705 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":71,"thread_start_us":71,"threads_started":1}
I20260927 02:50:04.075104 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.062628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39401 (local address 127.0.0.1:58472)
0927 02:50:04.062771 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:04.062834 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:04.062900 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:04.071108 (+ 8208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:04.071109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:04.071113 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:04.071181 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.071183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.072534 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.072535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.073899 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.073901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.074877 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.074878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.074944 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.074949 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:50:04.074980 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 02:50:04.075024 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":44,"thread_start_us":71,"threads_started":1}
I20260927 02:50:04.075496 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.062961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.0.98.2:46133)
0927 02:50:04.070917 (+ 7956us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:04.070922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:04.070934 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:50:04.070970 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:04.070972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:04.070990 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:04.071244 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:04.071320 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.072416 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.073759 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.074462 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.075143 (+ 681us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.075354 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:04.075382 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 02:50:04.075425 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7909,"thread_start_us":52,"threads_started":1}
I20260927 02:50:04.075580 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.062722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.0.98.1:39401)
0927 02:50:04.070934 (+ 8212us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:04.070937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:04.070947 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:50:04.070975 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:04.070976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:04.070990 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:04.071253 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:04.071320 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.072442 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.073982 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.074728 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.075330 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.075414 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:04.075453 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:50:04.075496 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8176,"thread_start_us":55,"threads_started":1}
I20260927 02:50:04.075786 13667 tablet_bootstrap.cc:492] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Bootstrap starting.
I20260927 02:50:04.076258 13300 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc6748322f6429c92079f279573c8c7 (DEFAULT_TABLE table=test-table [id=48a7643258f8425798444acfd0996bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:04.076555 13300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc6748322f6429c92079f279573c8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:04.076448 13434 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc6748322f6429c92079f279573c8c7 (DEFAULT_TABLE table=test-table [id=48a7643258f8425798444acfd0996bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:04.076763 13434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc6748322f6429c92079f279573c8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:04.079051 13668 tablet_bootstrap.cc:492] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73: Bootstrap starting.
I20260927 02:50:04.079874 13668 tablet_bootstrap.cc:654] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:04.080664 13668 log.cc:824] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:04.081027 13667 tablet_bootstrap.cc:654] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:04.081337 13667 log.cc:824] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:04.081720 13669 tablet_bootstrap.cc:492] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Bootstrap starting.
I20260927 02:50:04.082219 13667 tablet_bootstrap.cc:492] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: No bootstrap required, opened a new log
I20260927 02:50:04.082301 13667 ts_tablet_manager.cc:1397] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 02:50:04.084997 13667 raft_consensus.cc:348] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.085183 13667 raft_consensus.cc:374] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:04.085219 13667 raft_consensus.cc:729] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8baeb34efeea467f9e0f6be0672065e2, State: Initialized, Role: FOLLOWER
I20260927 02:50:04.085362 13667 consensus_queue.cc:260] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.085638 13667 ts_tablet_manager.cc:1428] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 02:50:04.085665 13634 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
I20260927 02:50:04.086287 13669 tablet_bootstrap.cc:654] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:04.086619 13669 log.cc:824] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:04.087529 13668 tablet_bootstrap.cc:492] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73: No bootstrap required, opened a new log
I20260927 02:50:04.087587 13669 tablet_bootstrap.cc:492] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: No bootstrap required, opened a new log
I20260927 02:50:04.087597 13668 ts_tablet_manager.cc:1397] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260927 02:50:04.087648 13669 ts_tablet_manager.cc:1397] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 02:50:04.090159 13668 raft_consensus.cc:348] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.090320 13668 raft_consensus.cc:374] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:04.090351 13668 raft_consensus.cc:729] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914369d36cf8481f9235fbb664b7cc73, State: Initialized, Role: FOLLOWER
I20260927 02:50:04.090468 13668 consensus_queue.cc:260] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.090719 13668 ts_tablet_manager.cc:1428] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:50:04.090860 13366 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
I20260927 02:50:04.090935 13669 raft_consensus.cc:348] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.091075 13669 raft_consensus.cc:374] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:04.091104 13669 raft_consensus.cc:729] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084bbf0940a04d8182bb5130407f541c, State: Initialized, Role: FOLLOWER
I20260927 02:50:04.091218 13669 consensus_queue.cc:260] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.091467 13669 ts_tablet_manager.cc:1428] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260927 02:50:04.091614 13500 heartbeater.cc:503] Master 127.0.98.62:36633 was elected leader, sending a full tablet report...
W20260927 02:50:04.097530 13501 tablet.cc:2369] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:50:04.099920 13675 raft_consensus.cc:482] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:04.100046 13675 raft_consensus.cc:504] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.100456 13675 leader_election.cc:288] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 914369d36cf8481f9235fbb664b7cc73 (127.0.98.1:39401), 8baeb34efeea467f9e0f6be0672065e2 (127.0.98.3:42181)
I20260927 02:50:04.104243 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.100677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48971 (local address 127.0.98.1:39401)
0927 02:50:04.100728 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:04.100731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:04.100899 (+ 168us) server_negotiation.cc:407] Connection header received
0927 02:50:04.100964 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:04.100965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:04.100977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:04.101020 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:04.101230 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.101958 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.103359 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.103908 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.103948 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.103959 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:04.104139 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:50:04.104175 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:50:04.104529 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.100715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38507 (local address 127.0.98.3:42181)
0927 02:50:04.100756 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:04.100758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:04.101029 (+ 271us) server_negotiation.cc:407] Connection header received
0927 02:50:04.101119 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:04.101120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:04.101129 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:04.101156 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:04.101356 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.102072 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.103502 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.104094 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.104150 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.104160 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:04.104403 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 02:50:04.104448 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:50:04.104837 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.100604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39401 (local address 127.0.98.2:48971)
0927 02:50:04.100824 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:04.100835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:04.100901 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:04.101068 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:04.101069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:04.101071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:04.101160 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.101163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.102050 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.102051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.103265 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.103268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.104010 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.104010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.104074 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.104078 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:04.104488 (+ 410us) client_negotiation.cc:240] Negotiation successful
0927 02:50:04.104533 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":128,"threads_started":1}
I20260927 02:50:04.104918 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.100606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42181 (local address 127.0.98.2:38507)
0927 02:50:04.100977 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:04.100985 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:04.101052 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:04.101216 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:04.101217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:04.101219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:04.101281 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.101283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.102166 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.102167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.103399 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:04.103402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.104240 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.104241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:04.104317 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.104320 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:50:04.104644 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 02:50:04.104690 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"spinlock_wait_cycles":11520,"thread_start_us":108,"threads_started":1}
I20260927 02:50:04.105278 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc6748322f6429c92079f279573c8c7" candidate_uuid: "084bbf0940a04d8182bb5130407f541c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8baeb34efeea467f9e0f6be0672065e2" is_pre_election: true
I20260927 02:50:04.105268 13320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc6748322f6429c92079f279573c8c7" candidate_uuid: "084bbf0940a04d8182bb5130407f541c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914369d36cf8481f9235fbb664b7cc73" is_pre_election: true
I20260927 02:50:04.105427 13588 raft_consensus.cc:2454] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 084bbf0940a04d8182bb5130407f541c in term 0.
I20260927 02:50:04.105428 13320 raft_consensus.cc:2454] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 084bbf0940a04d8182bb5130407f541c in term 0.
I20260927 02:50:04.105652 13389 leader_election.cc:302] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084bbf0940a04d8182bb5130407f541c, 914369d36cf8481f9235fbb664b7cc73; no voters:
I20260927 02:50:04.105968 13675 raft_consensus.cc:2781] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:50:04.106037 13675 raft_consensus.cc:482] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:50:04.106060 13675 raft_consensus.cc:3037] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:04.106998 13675 raft_consensus.cc:504] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.107131 13675 leader_election.cc:288] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [CANDIDATE]: Term 1 election: Requested vote from peers 914369d36cf8481f9235fbb664b7cc73 (127.0.98.1:39401), 8baeb34efeea467f9e0f6be0672065e2 (127.0.98.3:42181)
I20260927 02:50:04.107301 13588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc6748322f6429c92079f279573c8c7" candidate_uuid: "084bbf0940a04d8182bb5130407f541c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8baeb34efeea467f9e0f6be0672065e2"
I20260927 02:50:04.107362 13320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dc6748322f6429c92079f279573c8c7" candidate_uuid: "084bbf0940a04d8182bb5130407f541c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914369d36cf8481f9235fbb664b7cc73"
I20260927 02:50:04.107380 13588 raft_consensus.cc:3037] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:04.107434 13320 raft_consensus.cc:3037] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:04.108050 13320 raft_consensus.cc:2454] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 084bbf0940a04d8182bb5130407f541c in term 1.
I20260927 02:50:04.108227 13389 leader_election.cc:302] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 084bbf0940a04d8182bb5130407f541c, 914369d36cf8481f9235fbb664b7cc73; no voters:
I20260927 02:50:04.108256 13588 raft_consensus.cc:2454] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 084bbf0940a04d8182bb5130407f541c in term 1.
I20260927 02:50:04.108300 13675 raft_consensus.cc:2781] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:04.108438 13675 raft_consensus.cc:686] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 1 LEADER]: Becoming Leader. State: Replica: 084bbf0940a04d8182bb5130407f541c, State: Running, Role: LEADER
I20260927 02:50:04.108548 13675 consensus_queue.cc:237] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } }
I20260927 02:50:04.109345 13168 catalog_manager.cc:5495] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c reported cstate change: term changed from 0 to 1, leader changed from <none> to 084bbf0940a04d8182bb5130407f541c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "084bbf0940a04d8182bb5130407f541c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "084bbf0940a04d8182bb5130407f541c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 } health_report { overall_health: UNKNOWN } } }
I20260927 02:50:04.121078 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.118105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.0.98.2:46133)
0927 02:50:04.118161 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:04.118163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:04.118170 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:50:04.118210 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:04.118211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:04.118233 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:04.118308 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:50:04.118439 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.119215 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.120261 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:04.120547 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:04.120611 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:04.120754 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:50:04.120884 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:50:04.120959 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:04.120976 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 02:50:04.120996 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:50:04.123070 13320 raft_consensus.cc:1261] T 4dc6748322f6429c92079f279573c8c7 P 914369d36cf8481f9235fbb664b7cc73 [term 1 FOLLOWER]: Refusing update from remote peer 084bbf0940a04d8182bb5130407f541c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:04.123406 13588 raft_consensus.cc:1261] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 1 FOLLOWER]: Refusing update from remote peer 084bbf0940a04d8182bb5130407f541c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:04.123847 13678 consensus_queue.cc:1035] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8baeb34efeea467f9e0f6be0672065e2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:04.124022 13675 consensus_queue.cc:1035] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [LEADER]: Connected to new peer: Peer: permanent_uuid: "914369d36cf8481f9235fbb664b7cc73" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:04.127502 13684 mvcc.cc:205] Tried to move back new op lower bound from 7333795447284506624 to 7333795447229194240. Current Snapshot: MvccSnapshot[applied={T|T < 7333795447284506624}]
I20260927 02:50:04.128925 13682 mvcc.cc:205] Tried to move back new op lower bound from 7333795447284506624 to 7333795447229194240. Current Snapshot: MvccSnapshot[applied={T|T < 7333795447284506624}]
I20260927 02:50:04.129657 13168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:04.129757 13168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:04.130451 13168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccd551f5477246339d5932e0294d4aae: Sending DeleteTablet for 3 replicas of tablet 4dc6748322f6429c92079f279573c8c7
I20260927 02:50:04.130769 13434 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc6748322f6429c92079f279573c8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37032
I20260927 02:50:04.130844 13568 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc6748322f6429c92079f279573c8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48294
I20260927 02:50:04.131170 13300 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dc6748322f6429c92079f279573c8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58472
I20260927 02:50:04.131455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13206
I20260927 02:50:04.132941 13690 tablet_replica.cc:330] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: stopping tablet replica
I20260927 02:50:04.133059 13690 raft_consensus.cc:2229] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:50:04.133152 13690 raft_consensus.cc:2258] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:04.133531 13690 ts_tablet_manager.cc:1905] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:04.134974 13691 tablet_replica.cc:330] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: stopping tablet replica
I20260927 02:50:04.135437 13690 ts_tablet_manager.cc:1918] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:04.135514 13690 log.cc:1196] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/4dc6748322f6429c92079f279573c8c7
I20260927 02:50:04.135658 13691 raft_consensus.cc:2229] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:04.135754 13690 ts_tablet_manager.cc:1939] T 4dc6748322f6429c92079f279573c8c7 P 084bbf0940a04d8182bb5130407f541c: Deleting consensus metadata
I20260927 02:50:04.135893 13691 raft_consensus.cc:2258] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:04.136159 13155 catalog_manager.cc:4841] TS 084bbf0940a04d8182bb5130407f541c (127.0.98.2:46133): tablet 4dc6748322f6429c92079f279573c8c7 (table test-table [id=48a7643258f8425798444acfd0996bf5]) successfully deleted
I20260927 02:50:04.139416 13691 ts_tablet_manager.cc:1905] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:50:04.141112 13154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39401 (error 108)
I20260927 02:50:04.141316 13691 ts_tablet_manager.cc:1918] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:04.141449 13691 log.cc:1196] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/4dc6748322f6429c92079f279573c8c7
I20260927 02:50:04.141728 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:04.141617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39401 (local address 127.0.0.1:58478)
0927 02:50:04.141666 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:04.141694 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:50:04.141759 13691 ts_tablet_manager.cc:1939] T 4dc6748322f6429c92079f279573c8c7 P 8baeb34efeea467f9e0f6be0672065e2: Deleting consensus metadata
W20260927 02:50:04.141844 13154 catalog_manager.cc:4584] TS 914369d36cf8481f9235fbb664b7cc73 (127.0.98.1:39401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4dc6748322f6429c92079f279573c8c7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39401: connect: Connection refused (error 111)
I20260927 02:50:04.142352 13153 catalog_manager.cc:4841] TS 8baeb34efeea467f9e0f6be0672065e2 (127.0.98.3:42181): tablet 4dc6748322f6429c92079f279573c8c7 (table test-table [id=48a7643258f8425798444acfd0996bf5]) successfully deleted
I20260927 02:50:04.142387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13370
I20260927 02:50:04.150985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13503
I20260927 02:50:04.156774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13135
I20260927 02:50:04.162237 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 02:50:04.496848 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 02:50:04.496941 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 02:50:04.830888 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 02:50:04.831 UTC [12202] LOG: received smart shutdown request
2026-09-27 02:50:04.834 UTC [12202] LOG: background worker "logical replication launcher" (PID 12211) exited with exit code 1
2026-09-27 02:50:04.834 UTC [12206] LOG: shutting down
2026-09-27 02:50:04.979 UTC [12202] LOG: database system is shut down
2026-09-27T02:50:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45006 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:50:05 dist-test-slave-f4rg krb5kdc[13702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:50:05 dist-test-slave-f4rg krb5kdc[13702](info): set up 2 sockets
Sep 27 02:50:05 dist-test-slave-f4rg krb5kdc[13702](info): commencing operation
krb5kdc: starting...
W20260927 02:50:06.002405 392 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:50:06.111208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38201,127.0.98.61:38185,127.0.98.60:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:06.188354 13715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:06.188545 13715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:06.188572 13715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:06.189677 13715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:06.189713 13715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:06.189725 13715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:06.189749 13715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:06.189769 13715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:06.191262 13715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38201,127.0.98.61:38185,127.0.98.60:36523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:06.191481 13715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:06.191684 13715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:06.196372 13721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:06.196377 13723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:06.196367 13720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:06.196635 13715 server_base.cc:1034] running on GCE node
I20260927 02:50:06.196897 13715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:06.197187 13715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:06.198333 13715 hybrid_clock.cc:648] HybridClock initialized: now 1790477406198318 us; error 28 us; skew 500 ppm
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:06.201960 13715 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:06.202419 13715 webserver.cc:460] Webserver started at http://127.0.98.62:43617/ using document root <none> and password file <none>
I20260927 02:50:06.202670 13715 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:06.202728 13715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:06.202879 13715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:06.204200 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "a54aca11f48e4bd99492ecd94eb2de81"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "9e07c38021b0ca57151c0d6110be75d0"
server_key_iv: "663be936528b5eac278436551c01816f"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.204576 13715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "a54aca11f48e4bd99492ecd94eb2de81"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "9e07c38021b0ca57151c0d6110be75d0"
server_key_iv: "663be936528b5eac278436551c01816f"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.206148 13715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:50:06.206947 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:06.207103 13715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:06.207216 13715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "a54aca11f48e4bd99492ecd94eb2de81"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "9e07c38021b0ca57151c0d6110be75d0"
server_key_iv: "663be936528b5eac278436551c01816f"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.207314 13715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:06.239993 13715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:06.242704 13715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:06.242884 13715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:06.246735 13715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38201
I20260927 02:50:06.246876 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38201 every 8 connection(s)
I20260927 02:50:06.247095 13715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:06.247932 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:06.249416 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:06.255321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13715
I20260927 02:50:06.255400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:50:06.255563 392 external_mini_cluster.cc:1428] Setting key b42de9aa0b9ae07d3f36274b3a945ffa
I20260927 02:50:06.258395 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.250095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.0.98.62:38201)
0927 02:50:06.250882 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.250887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.250904 (+ 17us) server_negotiation.cc:407] Connection header received
0927 02:50:06.250971 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.250973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.250987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.251051 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.251588 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.252369 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.253210 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.253506 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.253577 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.255479 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.255483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.255484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.255489 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.257058 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.257403 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.257406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.257407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.257503 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.257708 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.257709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.257709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.257902 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.257962 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.258145 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.258237 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":75,"threads_started":1}
I20260927 02:50:06.258428 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.250075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38201 (local address 127.0.0.1:60784)
0927 02:50:06.250228 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.250262 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.250725 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.251091 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.251094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.251346 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.251510 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.251515 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.252437 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.252438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.253148 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.253151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.253560 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.253561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.253640 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.254046 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.254054 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.255370 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.257138 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.257144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.257146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.257332 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.257560 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.257561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.257561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.257654 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.257956 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.257961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.258030 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.258120 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.258317 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":41,"thread_start_us":81,"threads_started":1}
I20260927 02:50:06.259148 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:50:06.259469 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.259347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40672)
0927 02:50:06.259393 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.259444 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:50:06.259604 13731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
I20260927 02:50:06.260018 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.259934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40682)
0927 02:50:06.259976 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.260001 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 02:50:06.260115 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38185: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:50:06.278049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38185
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38201,127.0.98.61:38185,127.0.98.60:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:06.309333 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } attempt: 1
I20260927 02:50:06.309828 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.309664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40690)
0927 02:50:06.309732 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.309786 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:50:06.310179 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.310082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40704)
0927 02:50:06.310128 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.310158 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 02:50:06.310271 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38185: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
W20260927 02:50:06.357262 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:06.357453 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:06.357473 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:06.358608 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:06.358649 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:06.358661 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:06.358685 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:06.358701 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:06.360194 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38201,127.0.98.61:38185,127.0.98.60:36523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:06.360411 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:06.360607 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:06.365090 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:06.365198 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:06.365213 13798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:06.365273 13790 server_base.cc:1034] running on GCE node
I20260927 02:50:06.365564 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:06.365808 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:06.366971 13790 hybrid_clock.cc:648] HybridClock initialized: now 1790477406366954 us; error 26 us; skew 500 ppm
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:06.372723 13790 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:50:06.373184 13790 webserver.cc:460] Webserver started at http://127.0.98.61:37933/ using document root <none> and password file <none>
I20260927 02:50:06.373382 13790 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:06.373430 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:06.373565 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:06.374622 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data/instance:
uuid: "6348bceee05b48daacd3d881385c8c8b"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "df055014ff27966ba5aeb24999ca34c6"
server_key_iv: "05b669421f9e888b24136b3a96bba25a"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.374977 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/wal/instance:
uuid: "6348bceee05b48daacd3d881385c8c8b"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "df055014ff27966ba5aeb24999ca34c6"
server_key_iv: "05b669421f9e888b24136b3a96bba25a"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.376176 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:50:06.376878 13805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:06.377015 13790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:06.377095 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/wal
uuid: "6348bceee05b48daacd3d881385c8c8b"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "df055014ff27966ba5aeb24999ca34c6"
server_key_iv: "05b669421f9e888b24136b3a96bba25a"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.377156 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:06.378408 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } attempt: 2
I20260927 02:50:06.378767 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.378632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40718)
0927 02:50:06.378691 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.378731 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:50:06.379067 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.378973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40734)
0927 02:50:06.379012 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.379038 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:50:06.379155 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38185: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38185: connect: Connection refused (error 111)
I20260927 02:50:06.386356 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:06.388425 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:06.388572 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:06.392167 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38185
I20260927 02:50:06.392226 13857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38185 every 8 connection(s)
I20260927 02:50:06.392513 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:50:06.393306 13858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:06.394721 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:06.402380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13790
I20260927 02:50:06.402477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-1/wal/instance
I20260927 02:50:06.402530 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.395399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.0.98.62:38201)
0927 02:50:06.395449 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.395451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.395632 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:50:06.396069 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.396071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.396081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.396122 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.396670 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.397147 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.398063 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.398273 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.398330 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.400177 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.400179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.400179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.400185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.401411 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.401727 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.401728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.401728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.401776 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.401962 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.401963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.401963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.402063 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.402110 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.402380 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.402439 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:50:06.402650 392 external_mini_cluster.cc:1428] Setting key f52f7a3ed50dbc418f849863b3e01eec
I20260927 02:50:06.402741 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.395362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38201 (local address 127.0.0.1:60792)
0927 02:50:06.395509 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.395549 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.395975 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.396195 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.396198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.396425 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.396611 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.396614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.397213 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.397214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.397976 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.397977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.398342 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.398343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.398439 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.398881 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.398889 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.400100 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.401471 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.401475 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.401477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.401668 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.401825 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.401825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.401825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.401918 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.402108 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.402111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.402262 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.402371 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.402524 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260927 02:50:06.403272 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 02:50:06.409734 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.403481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40744)
0927 02:50:06.403523 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.403533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.403568 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.403783 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.403783 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.403904 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.403944 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.403945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.404733 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.404734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.405223 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.405224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.405638 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.405639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.405701 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.406052 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.406056 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.407330 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.408860 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.408861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.408861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.409044 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.409231 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.409232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.409232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.409294 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.409512 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.409513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.409547 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.409592 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.409663 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:50:06.409825 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.403533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.98.61:38185)
0927 02:50:06.403670 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.403674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.403689 (+ 15us) server_negotiation.cc:407] Connection header received
0927 02:50:06.403704 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.403705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.403715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.403775 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.404002 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.404669 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.405285 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.405581 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.405636 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.407408 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.407416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.407416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.407421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.408783 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.409122 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.409125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.409125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.409179 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.409346 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.409346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.409346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.409466 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.409519 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.409624 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.409717 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260927 02:50:06.410404 13858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:50:06.410679 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.410591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46106)
0927 02:50:06.410619 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.410662 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 02:50:06.410801 13807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
I20260927 02:50:06.411275 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.411154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46114)
0927 02:50:06.411216 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.411250 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 02:50:06.411401 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:50:06.423682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36523
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38201,127.0.98.61:38185,127.0.98.60:36523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:06.460646 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } attempt: 1
I20260927 02:50:06.461186 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.461061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46130)
0927 02:50:06.461118 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.461153 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 02:50:06.461562 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.461439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46138)
0927 02:50:06.461481 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.461542 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:50:06.461664 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
I20260927 02:50:06.470301 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } attempt: 3
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:50:06.477415 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.470670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40752)
0927 02:50:06.470751 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.470764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.470861 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.471013 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.471014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.471155 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.471225 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.471227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.472150 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.472151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.472700 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.472702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.473172 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.473173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.473253 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.473701 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.473707 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.475013 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.476566 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.476567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.476567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.476723 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.476923 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.476924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.476924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.476978 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.477195 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.477195 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.477232 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.477276 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.477346 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 02:50:06.477463 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.470759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.0.98.61:38185)
0927 02:50:06.470845 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.470847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.470859 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:50:06.470928 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.470929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.470942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.470994 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.471286 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.472067 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.472777 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.473085 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.473154 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.475091 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.475093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.475093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.475100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.476482 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.476783 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.476784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.476784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.476859 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.477026 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.477027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.477027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.477147 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.477202 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.477307 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.477378 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:50:06.477960 13783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:50:06.478264 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.478178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46146)
0927 02:50:06.478215 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.478242 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 02:50:06.478534 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.478474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46150)
0927 02:50:06.478501 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.478520 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 02:50:06.478613 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
W20260927 02:50:06.503583 13865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:06.503785 13865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:06.503813 13865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:06.504906 13865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:06.504942 13865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:06.504961 13865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:06.504976 13865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:06.504992 13865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:06.506458 13865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38201,127.0.98.61:38185,127.0.98.60:36523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:06.506682 13865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:06.506908 13865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:50:06.509763 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } attempt: 1
I20260927 02:50:06.510200 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.510069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46160)
0927 02:50:06.510122 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.510166 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:50:06.510561 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.510464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46172)
0927 02:50:06.510499 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.510534 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:50:06.510650 13783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
W20260927 02:50:06.511340 13871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:06.511364 13873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:06.511371 13870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:06.511670 13865 server_base.cc:1034] running on GCE node
I20260927 02:50:06.511914 13865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:06.512167 13865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:06.513299 13865 hybrid_clock.cc:648] HybridClock initialized: now 1790477406513285 us; error 28 us; skew 500 ppm
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:06.515623 13865 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:50:06.516059 13865 webserver.cc:460] Webserver started at http://127.0.98.60:34541/ using document root <none> and password file <none>
I20260927 02:50:06.516245 13865 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:06.516291 13865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:06.516415 13865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:06.517449 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data/instance:
uuid: "c1b12bfeca02406ab604d3d6a73f8a8c"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "25c9bd031d2735c3d52b33844efa6c2d"
server_key_iv: "a746ddf9fe82ea42bcbde21c9c468074"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.517782 13865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/wal/instance:
uuid: "c1b12bfeca02406ab604d3d6a73f8a8c"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "25c9bd031d2735c3d52b33844efa6c2d"
server_key_iv: "a746ddf9fe82ea42bcbde21c9c468074"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.518996 13865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:50:06.519636 13880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:06.519783 13865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:06.519860 13865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/wal
uuid: "c1b12bfeca02406ab604d3d6a73f8a8c"
format_stamp: "Formatted at 2026-09-27 02:50:06 on dist-test-slave-f4rg"
server_key: "25c9bd031d2735c3d52b33844efa6c2d"
server_key_iv: "a746ddf9fe82ea42bcbde21c9c468074"
server_key_version: "encryptionkey@0"
I20260927 02:50:06.519917 13865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:06.529824 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } attempt: 2
I20260927 02:50:06.530228 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.530117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46184)
0927 02:50:06.530169 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.530201 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:50:06.530522 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.530444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46196)
0927 02:50:06.530478 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.530500 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:50:06.530596 13858 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36523: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36523: connect: Connection refused (error 111)
I20260927 02:50:06.532076 13865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:06.533769 13865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:06.533893 13865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:06.537470 13865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36523
I20260927 02:50:06.537499 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36523 every 8 connection(s)
I20260927 02:50:06.537838 13865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:50:06.538606 13933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:06.540073 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:06.547835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 13865
I20260927 02:50:06.547909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790477232139228-392-0/minicluster-data/master-2/wal/instance
I20260927 02:50:06.548003 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.540705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.0.98.62:38201)
0927 02:50:06.540756 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.540758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.540905 (+ 147us) server_negotiation.cc:407] Connection header received
0927 02:50:06.541366 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.541368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.541380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.541423 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.542008 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.542474 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.543433 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.543615 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.543678 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.545564 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.545565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.545566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.545573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.546789 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.547160 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.547161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.547162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.547214 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.547418 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.547419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.547419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.547536 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.547589 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.547836 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.547914 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:50:06.548063 392 external_mini_cluster.cc:1428] Setting key 0fe39729370d1fe9ff0119ae64d04607
I20260927 02:50:06.548180 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.540660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38201 (local address 127.0.0.1:60794)
0927 02:50:06.540801 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.540834 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.541274 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.541504 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.541507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.541747 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.541929 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.541932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.542548 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.542549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.543360 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.543362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.543686 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.543687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.543800 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.544261 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.544269 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.545482 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.546881 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.546887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.546890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.547096 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.547266 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.547267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.547267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.547366 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.547589 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.547595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.547721 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.547805 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.547947 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260927 02:50:06.548795 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:50:06.555124 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.549093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.98.61:38185)
0927 02:50:06.549137 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.549139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.549147 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:06.549171 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.549173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.549182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.549243 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.549454 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.549956 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.550769 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.551090 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.551157 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.552851 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.552853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.552853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.552860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.554159 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.554452 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.554453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.554454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.554502 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.554655 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.554656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.554657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.554765 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.554847 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.555002 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.555062 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:50:06.555164 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.549021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38185 (local address 127.0.0.1:40754)
0927 02:50:06.549058 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.549067 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.549101 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.549247 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.549247 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.549353 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.549397 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.549398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.550018 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.550019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.550704 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.550706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.551150 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.551151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.551212 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.551529 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.551533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.552777 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.554244 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.554245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.554245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.554396 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.554557 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.554557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.554558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.554609 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.554856 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.554858 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.554902 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.554980 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.555057 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":72}
I20260927 02:50:06.555603 13933 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:50:06.556486 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.548386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.0.98.60:36523)
0927 02:50:06.548521 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.548526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.548633 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:50:06.548680 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.548683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.548696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.548759 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.549587 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.550454 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.551337 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.551715 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.551781 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.553880 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.553888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.553888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.553895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.555389 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.555749 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.555752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.555752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.555808 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.556000 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.556001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.556001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.556130 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.556287 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.556328 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.556394 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:06.562930 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.555848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46214)
0927 02:50:06.555987 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.555996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.556032 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.556148 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.556148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.556259 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.556284 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.556285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.556929 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.556930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.557569 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.557571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.557888 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.557889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.557932 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.558243 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.558247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.559421 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.561457 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.561458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.561458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.561615 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.561825 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.561825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.561826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.561883 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.562608 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.562608 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.562640 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.562690 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.562774 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 02:50:06.563084 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.555887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.0.98.60:36523)
0927 02:50:06.556009 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.556011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.556021 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:50:06.556071 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.556071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.556080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.556135 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.556340 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.556868 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.557643 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.557835 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.557886 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.559500 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.559501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.559502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.559508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.561396 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.561710 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.561710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.561711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.561761 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.561939 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.561939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.561940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.562048 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.562106 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.562732 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.562841 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260927 02:50:06.563568 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.557331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.0.98.62:38201)
0927 02:50:06.557377 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.557379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.557406 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:50:06.557450 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.557451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.557459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.557514 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.557904 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.558344 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.559208 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.559412 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.559469 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.561338 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.561339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.561339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.561345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.562537 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.562830 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.562831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.562831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.562885 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.563049 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.563051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.563051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.563192 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.563381 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.563416 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.563482 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:50:06.564518 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c: Bootstrap starting.
I20260927 02:50:06.565231 13933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:06.565507 13933 log.cc:824] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:06.566236 13933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c: No bootstrap required, opened a new log
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:50:06.568056 13933 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.568225 13933 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:06.568270 13933 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b12bfeca02406ab604d3d6a73f8a8c, State: Initialized, Role: FOLLOWER
I20260927 02:50:06.568398 13933 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.568671 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.568727 13933 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:06.568773 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:50:06.570784 13951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:50:06.570863 13951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:50:06.571626 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.564101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.0.98.61:38185)
0927 02:50:06.564139 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.564141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.564148 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:50:06.564168 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.564169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.564178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.564237 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.564599 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.565237 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.565822 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.566059 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.566113 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.568355 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.568357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.568357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.568364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.569972 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.570323 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.570324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.570324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.570375 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.571096 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.571097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.571097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.571238 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.571409 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.571448 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.571528 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:50:06.572937 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:50:06.571218 13865 master_runner.cc:186] Error getting master registration for 127.0.98.62:38201: OK, instance_id { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" instance_seqno: 1790477406245645 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:50:06.573235 13865 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:50:06.581496 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.574392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.0.98.62:38201)
0927 02:50:06.574427 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.574428 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.574435 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:50:06.574467 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.574467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.574476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.574528 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.575222 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.575951 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.576951 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.577250 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.577291 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.578832 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.578833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.578833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.578840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.580336 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.580698 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.580699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.580699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.580753 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.580945 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.580945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.580946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.581065 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.581134 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.581333 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.581405 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:50:06.581709 13861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.574216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.0.98.61:38185)
0927 02:50:06.574359 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.574361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.574619 (+ 258us) server_negotiation.cc:407] Connection header received
0927 02:50:06.574652 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.574653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.574662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.574706 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.575308 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.575999 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.576950 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.577243 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.577277 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.578915 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.578916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.578917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.578923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.580491 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.580907 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.580909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.580909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.580961 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.581169 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.581169 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.581170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.581287 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.581365 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.581557 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.581621 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 02:50:06.582628 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.574400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.98.60:36523)
0927 02:50:06.574435 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.574437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.574940 (+ 503us) server_negotiation.cc:407] Connection header received
0927 02:50:06.575029 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.575030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.575040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.575067 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.575717 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.576395 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.577264 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.577918 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.577959 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.579994 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.579995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.579996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.580003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.581505 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.581859 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.581860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.581860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.581911 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.582101 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.582101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.582102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.582221 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.582284 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.582476 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.582545 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:50:06.585927 13783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } attempt: 2
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:50:06.592018 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.586208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.98.60:36523)
0927 02:50:06.586239 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.586241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.586249 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:06.586306 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.586306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.586313 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.586351 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.586595 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.587071 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.587986 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.588162 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.588201 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.589800 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.589802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.589802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.589808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.591026 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.591370 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.591371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.591372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.591431 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.591617 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.591618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.591618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.591723 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.591770 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.591908 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.591960 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:50:06.592072 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.586143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46230)
0927 02:50:06.586200 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.586211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.586252 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.586357 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.586357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.586503 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.586540 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.586541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.587128 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.587129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.587919 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.587921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.588217 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.588218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.588272 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.588585 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.588589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.589729 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.591099 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.591100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.591100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.591304 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.591505 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.591506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.591506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.591564 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.591779 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.591780 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.591822 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.591892 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.591975 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:50:06.593303 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: Bootstrap starting.
I20260927 02:50:06.593883 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:06.594128 13783 log.cc:824] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:06.594779 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: No bootstrap required, opened a new log
I20260927 02:50:06.596177 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.596284 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:06.596315 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a54aca11f48e4bd99492ecd94eb2de81, State: Initialized, Role: FOLLOWER
I20260927 02:50:06.596432 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.596706 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:06.596686 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.596812 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:50:06.598311 13972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:50:06.598362 13972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:50:06.599021 13715 master_runner.cc:186] Error getting master registration for 127.0.98.61:38185: OK, instance_id { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" instance_seqno: 1790477406391155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:50:06.601630 13715 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:50:06.621737 13858 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } attempt: 3
Sep 27 02:50:06 dist-test-slave-f4rg krb5kdc[13702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:50:06.629787 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.622052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36523 (local address 127.0.0.1:46232)
0927 02:50:06.622102 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:06.622113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:06.622161 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:06.622418 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:06.622419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:06.622605 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:06.622659 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.622661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.623613 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.623614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.624416 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:06.624419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.624871 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.624872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:06.624932 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.625432 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:06.625437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:06.626938 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:06.628673 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.628675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.628675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.628889 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.629123 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:06.629124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:06.629125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:06.629199 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:06.629469 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:06.629469 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:06.629516 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 02:50:06.629584 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 02:50:06.629692 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:50:06.629787 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:06.622242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.0.98.60:36523)
0927 02:50:06.622285 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:06.622286 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:06.622293 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:50:06.622314 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:06.622315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:06.622324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:06.622366 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:06.622739 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.623542 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.624497 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:06.624799 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:06.624863 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:06.627021 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:06.627023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:06.627023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:06.627030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:06.628596 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.628958 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.628959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.628959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.629015 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:06.629260 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:06.629262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:06.629262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:06.629405 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:06.629472 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:06.629613 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:50:06.629688 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:50:06.632279 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: Bootstrap starting.
I20260927 02:50:06.632967 13858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:06.633267 13858 log.cc:824] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:06.634081 13858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: No bootstrap required, opened a new log
I20260927 02:50:06.635751 13940 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:06.635840 13940 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.636070 13940 leader_election.cc:288] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a54aca11f48e4bd99492ecd94eb2de81 (127.0.98.62:38201), 6348bceee05b48daacd3d881385c8c8b (127.0.98.61:38185)
I20260927 02:50:06.636009 13858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.636405 13858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:06.636404 13757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54aca11f48e4bd99492ecd94eb2de81" is_pre_election: true
I20260927 02:50:06.636469 13858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6348bceee05b48daacd3d881385c8c8b, State: Initialized, Role: FOLLOWER
I20260927 02:50:06.636540 13757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b12bfeca02406ab604d3d6a73f8a8c in term 0.
I20260927 02:50:06.636650 13858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.636895 13883 leader_election.cc:302] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a54aca11f48e4bd99492ecd94eb2de81, c1b12bfeca02406ab604d3d6a73f8a8c; no voters:
I20260927 02:50:06.637073 13940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:50:06.637120 13940 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:50:06.637140 13940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:06.637085 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6348bceee05b48daacd3d881385c8c8b" is_pre_election: true
I20260927 02:50:06.637192 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1b12bfeca02406ab604d3d6a73f8a8c in term 0.
I20260927 02:50:06.637392 13858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:06.637862 13940 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.637969 13940 leader_election.cc:288] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [CANDIDATE]: Term 1 election: Requested vote from peers a54aca11f48e4bd99492ecd94eb2de81 (127.0.98.62:38201), 6348bceee05b48daacd3d881385c8c8b (127.0.98.61:38185)
I20260927 02:50:06.638238 13757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a54aca11f48e4bd99492ecd94eb2de81"
I20260927 02:50:06.638304 13757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:06.639127 13757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b12bfeca02406ab604d3d6a73f8a8c in term 1.
I20260927 02:50:06.639524 13883 leader_election.cc:302] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a54aca11f48e4bd99492ecd94eb2de81, c1b12bfeca02406ab604d3d6a73f8a8c; no voters:
W20260927 02:50:06.639305 13811 tablet.cc:2369] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:50:06.639351 13986 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:50:06.639611 13986 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:50:06.639742 13940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:06.639864 13940 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [term 1 LEADER]: Becoming Leader. State: Replica: c1b12bfeca02406ab604d3d6a73f8a8c, State: Running, Role: LEADER
I20260927 02:50:06.639937 13940 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } }
I20260927 02:50:06.639993 13832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6348bceee05b48daacd3d881385c8c8b"
I20260927 02:50:06.640049 13832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:06.640262 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b12bfeca02406ab604d3d6a73f8a8c. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.640326 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:06.640599 13990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:06.640801 13990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:06.640836 13832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1b12bfeca02406ab604d3d6a73f8a8c in term 1.
I20260927 02:50:06.643638 13832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [term 1 FOLLOWER]: Refusing update from remote peer c1b12bfeca02406ab604d3d6a73f8a8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:06.643963 13940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:06.645491 13757 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [term 1 FOLLOWER]: Refusing update from remote peer c1b12bfeca02406ab604d3d6a73f8a8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:06.645783 13940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:06.647446 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b12bfeca02406ab604d3d6a73f8a8c. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.647547 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:50:06.648705 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.648792 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:06.649289 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.649384 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:50:06.649609 13991 mvcc.cc:205] Tried to move back new op lower bound from 7333795457607503872 to 7333795457597698048. Current Snapshot: MvccSnapshot[applied={T|T < 7333795457607503872}]
I20260927 02:50:06.649740 13995 mvcc.cc:205] Tried to move back new op lower bound from 7333795457607503872 to 7333795457597698048. Current Snapshot: MvccSnapshot[applied={T|T < 7333795457607503872}]
I20260927 02:50:06.650527 13940 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.650600 13940 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:06.651026 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.651115 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:50:06.651171 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b12bfeca02406ab604d3d6a73f8a8c. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.651216 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:50:06.651242 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a54aca11f48e4bd99492ecd94eb2de81" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38201 } } peers { permanent_uuid: "6348bceee05b48daacd3d881385c8c8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38185 } } peers { permanent_uuid: "c1b12bfeca02406ab604d3d6a73f8a8c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36523 } } }
I20260927 02:50:06.651284 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:50:06.651717 13993 mvcc.cc:205] Tried to move back new op lower bound from 7333795457607503872 to 7333795457597698048. Current Snapshot: MvccSnapshot[applied={T|T < 7333795457607503872}]
I20260927 02:50:06.652622 13990 catalog_manager.cc:1347] Generated new cluster ID: a766f143b0024b6a8bb7f9ab154416fc
I20260927 02:50:06.652668 13990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:50:06.676011 13990 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:50:06.676812 13990 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:06.683269 13990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1b12bfeca02406ab604d3d6a73f8a8c: Generated new TSK 0
I20260927 02:50:06.683539 13990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:50:07.598755 13972 catalog_manager.cc:1259] Loaded cluster ID: a766f143b0024b6a8bb7f9ab154416fc
I20260927 02:50:07.598842 13972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: loading cluster ID for follower catalog manager: success
I20260927 02:50:07.599632 13972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: acquiring CA information for follower catalog manager: success
I20260927 02:50:07.599943 13972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a54aca11f48e4bd99492ecd94eb2de81: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:50:07.640033 13986 catalog_manager.cc:1259] Loaded cluster ID: a766f143b0024b6a8bb7f9ab154416fc
I20260927 02:50:07.640106 13986 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: loading cluster ID for follower catalog manager: success
I20260927 02:50:07.640915 13986 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: acquiring CA information for follower catalog manager: success
I20260927 02:50:07.641209 13986 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6348bceee05b48daacd3d881385c8c8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:50:07.732721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13715
I20260927 02:50:07.737340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13790
I20260927 02:50:07.741348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 13865
2026-09-27T02:50:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2677 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:50:07 dist-test-slave-f4rg krb5kdc[14008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:50:07 dist-test-slave-f4rg krb5kdc[14008](info): set up 2 sockets
Sep 27 02:50:07 dist-test-slave-f4rg krb5kdc[14008](info): commencing operation
krb5kdc: starting...
W20260927 02:50:08.682528 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:50:08 dist-test-slave-f4rg krb5kdc[14008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:50:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:50:08.789906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:08.867466 14021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:08.867702 14021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:08.867723 14021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:08.868863 14021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:08.868917 14021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:08.868933 14021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:08.868944 14021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:08.868955 14021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:08.870445 14021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:08.870671 14021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:08.870927 14021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:08.875473 14026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:08.875460 14029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:08.875460 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:08.875485 14021 server_base.cc:1034] running on GCE node
I20260927 02:50:08.875865 14021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:08.876128 14021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:08.877332 14021 hybrid_clock.cc:648] HybridClock initialized: now 1790477408877320 us; error 32 us; skew 500 ppm
Sep 27 02:50:08 dist-test-slave-f4rg krb5kdc[14008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:08.880244 14021 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:08.880715 14021 webserver.cc:460] Webserver started at http://127.0.98.62:34591/ using document root <none> and password file <none>
I20260927 02:50:08.880995 14021 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:08.881057 14021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:08.881201 14021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:08.882437 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "3cb801b81c544051b734d6cf73913907"
format_stamp: "Formatted at 2026-09-27 02:50:08 on dist-test-slave-f4rg"
server_key: "3cdc3cbd0601d8a852faaf75c5682d38"
server_key_iv: "a87d27cb45558b86ec6f4b3a83cc1a5e"
server_key_version: "encryptionkey@0"
I20260927 02:50:08.882727 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cb801b81c544051b734d6cf73913907"
format_stamp: "Formatted at 2026-09-27 02:50:08 on dist-test-slave-f4rg"
server_key: "3cdc3cbd0601d8a852faaf75c5682d38"
server_key_iv: "a87d27cb45558b86ec6f4b3a83cc1a5e"
server_key_version: "encryptionkey@0"
I20260927 02:50:08.883945 14021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:50:08.884583 14036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:08.884712 14021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:08.884794 14021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "3cb801b81c544051b734d6cf73913907"
format_stamp: "Formatted at 2026-09-27 02:50:08 on dist-test-slave-f4rg"
server_key: "3cdc3cbd0601d8a852faaf75c5682d38"
server_key_iv: "a87d27cb45558b86ec6f4b3a83cc1a5e"
server_key_version: "encryptionkey@0"
I20260927 02:50:08.884860 14021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:08.896593 14021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:08.898242 14021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:08.898360 14021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:08.901924 14021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45421
I20260927 02:50:08.901957 14088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45421 every 8 connection(s)
I20260927 02:50:08.902258 14021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:08.903079 14089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:08.904018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14021
I20260927 02:50:08.904093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:50:08.904264 392 external_mini_cluster.cc:1428] Setting key 16f616972c2bf28278d0855fef420712
I20260927 02:50:08.905560 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907: Bootstrap starting.
I20260927 02:50:08.906247 14089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:08.906523 14089 log.cc:824] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:08.907182 14089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907: No bootstrap required, opened a new log
I20260927 02:50:08.908984 14089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb801b81c544051b734d6cf73913907" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260927 02:50:08.909124 14089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:08.909168 14089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cb801b81c544051b734d6cf73913907, State: Initialized, Role: FOLLOWER
I20260927 02:50:08.909296 14089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb801b81c544051b734d6cf73913907" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260927 02:50:08.909369 14089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:08.909400 14089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:08.909446 14089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:08.909950 14089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb801b81c544051b734d6cf73913907" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260927 02:50:08.910043 14089 leader_election.cc:302] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cb801b81c544051b734d6cf73913907; no voters:
I20260927 02:50:08.910205 14089 leader_election.cc:288] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:50:08.910256 14094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 02:50:08 dist-test-slave-f4rg krb5kdc[14008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:08.910430 14094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [term 1 LEADER]: Becoming Leader. State: Replica: 3cb801b81c544051b734d6cf73913907, State: Running, Role: LEADER
I20260927 02:50:08.910431 14089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:08.910589 14094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb801b81c544051b734d6cf73913907" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } }
I20260927 02:50:08.910951 14094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cb801b81c544051b734d6cf73913907. Latest consensus state: current_term: 1 leader_uuid: "3cb801b81c544051b734d6cf73913907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb801b81c544051b734d6cf73913907" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } } }
I20260927 02:50:08.910974 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cb801b81c544051b734d6cf73913907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cb801b81c544051b734d6cf73913907" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45421 } } }
I20260927 02:50:08.911037 14094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:08.911043 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907 [sys.catalog]: This master's current role is: LEADER
W20260927 02:50:08.912241 14109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:50:08.912290 14109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:50:08.912401 14101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:08.912583 14101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:08.914170 14101 catalog_manager.cc:1347] Generated new cluster ID: bacb5d30fa61410486c22755a56200b6
I20260927 02:50:08.914218 14101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:50:08.914723 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:08.904702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.0.98.62:45421)
0927 02:50:08.904844 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:08.904852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:08.904875 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:50:08.905364 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:08.905371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:08.905421 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:08.905531 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:08.906274 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:08.907280 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:08.907994 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:08.908399 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:08.908490 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:08.910900 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:08.910904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:08.910905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:08.910916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:08.912996 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:08.913410 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:08.913413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:08.913413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:08.913510 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:08.913691 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:08.913692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:08.913692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:08.913948 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:08.914101 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:08.914406 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 02:50:08.914529 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260927 02:50:08.915614 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:50:08.923463 14101 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:50:08.924010 14101 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:08.935159 14101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cb801b81c544051b734d6cf73913907: Generated new TSK 0
I20260927 02:50:08.935339 14101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:50:08.969180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14021
2026-09-27T02:50:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1227 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:50:09 dist-test-slave-f4rg krb5kdc[14116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:50:09 dist-test-slave-f4rg krb5kdc[14116](info): set up 2 sockets
Sep 27 02:50:09 dist-test-slave-f4rg krb5kdc[14116](info): commencing operation
krb5kdc: starting...
W20260927 02:50:09.914568 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:50:09 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:50:10.025125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:10.104130 14129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:10.104339 14129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:10.104369 14129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:10.105607 14129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:10.105657 14129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:10.105674 14129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:10.105691 14129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:10.105705 14129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:10.107216 14129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:10.107457 14129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:10.107686 14129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:10.112214 14137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.112231 14135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.112206 14134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.112401 14129 server_base.cc:1034] running on GCE node
I20260927 02:50:10.112763 14129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:10.113035 14129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:10.114212 14129 hybrid_clock.cc:648] HybridClock initialized: now 1790477410114196 us; error 28 us; skew 500 ppm
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:10.116760 14129 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:10.117208 14129 webserver.cc:460] Webserver started at http://127.0.98.62:33903/ using document root <none> and password file <none>
I20260927 02:50:10.117399 14129 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:10.117446 14129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:10.117564 14129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:10.118582 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "c9fc18798e07479687455e9137520fdf"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "189504672861f1ca5339d44a36d151db"
server_key_iv: "b3856126ebaec6cfd5f8d32fb0574a64"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.118927 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9fc18798e07479687455e9137520fdf"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "189504672861f1ca5339d44a36d151db"
server_key_iv: "b3856126ebaec6cfd5f8d32fb0574a64"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.120105 14129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:10.120790 14144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.120935 14129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:10.121016 14129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "c9fc18798e07479687455e9137520fdf"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "189504672861f1ca5339d44a36d151db"
server_key_iv: "b3856126ebaec6cfd5f8d32fb0574a64"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.121075 14129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:10.136600 14129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:10.138370 14129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:10.138509 14129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:10.142208 14129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34859
I20260927 02:50:10.142243 14196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34859 every 8 connection(s)
I20260927 02:50:10.142567 14129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:10.143402 14197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:10.145619 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Bootstrap starting.
I20260927 02:50:10.146126 14197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:10.146350 14197 log.cc:824] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:10.146988 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: No bootstrap required, opened a new log
I20260927 02:50:10.148502 14197 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.148640 14197 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:10.148662 14197 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fc18798e07479687455e9137520fdf, State: Initialized, Role: FOLLOWER
I20260927 02:50:10.148761 14197 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.148814 14197 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:10.148840 14197 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:10.148877 14197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:10.149494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14129
I20260927 02:50:10.149597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:50:10.149814 392 external_mini_cluster.cc:1428] Setting key 32bf2e4d024bdbe07913fe601cfb7bf1
I20260927 02:50:10.149786 14197 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.149909 14197 leader_election.cc:302] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9fc18798e07479687455e9137520fdf; no voters:
I20260927 02:50:10.150409 14197 leader_election.cc:288] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:50:10.150503 14202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:10.150676 14197 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:10.150684 14202 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 LEADER]: Becoming Leader. State: Replica: c9fc18798e07479687455e9137520fdf, State: Running, Role: LEADER
I20260927 02:50:10.150933 14202 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.151773 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9fc18798e07479687455e9137520fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260927 02:50:10.151862 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:10.152051 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9fc18798e07479687455e9137520fdf. Latest consensus state: current_term: 1 leader_uuid: "c9fc18798e07479687455e9137520fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260927 02:50:10.152036 14216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:10.152279 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:10.152689 14216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:10.154127 14216 catalog_manager.cc:1347] Generated new cluster ID: 9ff8d8a352f04f9b8941d61fd1633218
I20260927 02:50:10.154181 14216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:10.159749 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.150257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.98.62:34859)
0927 02:50:10.150431 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.150441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.150476 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:50:10.150598 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.150606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.150665 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.150845 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:10.151569 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.152960 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.153947 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.154448 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.154557 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.156768 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:10.156772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:10.156773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:10.156785 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:10.158312 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.158618 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.158621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.158621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.158714 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.158933 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.158934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.158935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.159200 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:10.159346 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.159448 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.159582 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260927 02:50:10.162212 14216 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:50:10.162726 14216 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:10.170012 14216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Generated new TSK 0
I20260927 02:50:10.170187 14216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:50:10.197152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:10.276171 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:10.276377 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:10.276409 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:10.277541 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:10.277590 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:10.277626 14223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:50:10.279105 14223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:10.279336 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:10.279539 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:10.284219 14229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.284246 14231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.284304 14223 server_base.cc:1034] running on GCE node
W20260927 02:50:10.284219 14228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.284632 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:10.284857 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:10.285985 14223 hybrid_clock.cc:648] HybridClock initialized: now 1790477410285973 us; error 25 us; skew 500 ppm
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:10.288333 14223 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:50:10.288724 14223 webserver.cc:460] Webserver started at http://127.0.98.1:42339/ using document root <none> and password file <none>
I20260927 02:50:10.288909 14223 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:10.288947 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:10.289036 14223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:10.290004 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a1b7b98551e431383cc1c9696b9e389"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "6bc4ad83d4030dab1747cbb088690425"
server_key_iv: "4b16787167e5feec40f58ac25e3aa3ee"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.290289 14223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a1b7b98551e431383cc1c9696b9e389"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "6bc4ad83d4030dab1747cbb088690425"
server_key_iv: "4b16787167e5feec40f58ac25e3aa3ee"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.291546 14223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:50:10.292238 14238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.292418 14223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:10.292500 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "6a1b7b98551e431383cc1c9696b9e389"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "6bc4ad83d4030dab1747cbb088690425"
server_key_iv: "4b16787167e5feec40f58ac25e3aa3ee"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.292567 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:10.305512 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:10.307582 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:10.307713 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:10.307909 14223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:10.308174 14223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:10.308205 14223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.308226 14223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:10.308250 14223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.313508 14223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36257
I20260927 02:50:10.313558 14351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36257 every 8 connection(s)
I20260927 02:50:10.313843 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:10.321866 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.314452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42027 (local address 127.0.98.62:34859)
0927 02:50:10.314518 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.314519 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.314709 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:50:10.315211 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.315212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.315222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.315281 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:10.316124 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.316613 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.317522 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.317718 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.317760 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.319639 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:10.319640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:10.319640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:10.319647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:10.320871 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.321174 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.321175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.321175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.321222 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.321387 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.321387 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.321387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.321480 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:10.321522 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.321717 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.321777 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:50:10.321987 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.314421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:42027)
0927 02:50:10.314572 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.314611 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.315124 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.315340 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.315343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.315864 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:10.316055 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.316060 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.316673 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.316673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.317458 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.317460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.317771 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.317771 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.317864 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.318292 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:10.318300 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:10.319558 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:10.320928 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.320931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.320933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.321116 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.321266 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.321267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.321267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.321348 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.321518 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:10.321521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:10.321627 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.321698 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.321809 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260927 02:50:10.321975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14223
I20260927 02:50:10.322070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:50:10.322266 392 external_mini_cluster.cc:1428] Setting key 41ee87a9fe2927813d6de19aa2432e0f
I20260927 02:50:10.322449 14352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260927 02:50:10.322532 14352 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:10.322939 14352 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260927 02:50:10.323495 14161 ts_manager.cc:194] Registered new tserver with Master: 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1:36257)
I20260927 02:50:10.324296 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42027
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:50:10.343364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:10.421370 14357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:10.421566 14357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:10.421588 14357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:10.422734 14357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:10.422844 14357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:10.422900 14357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:50:10.424497 14357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:10.424724 14357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:10.424925 14357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:10.429711 14363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.429735 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.429679 14365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.429828 14357 server_base.cc:1034] running on GCE node
I20260927 02:50:10.430060 14357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:10.430299 14357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:10.431443 14357 hybrid_clock.cc:648] HybridClock initialized: now 1790477410431429 us; error 28 us; skew 500 ppm
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:10.433709 14357 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:50:10.434077 14357 webserver.cc:460] Webserver started at http://127.0.98.2:34125/ using document root <none> and password file <none>
I20260927 02:50:10.434252 14357 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:10.434290 14357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:10.434377 14357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:10.435415 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "9d6c2e9a60f44aea9572361308f2cd2e"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "7088a95a46f018c9cba428adf6f51ed2"
server_key_iv: "23de1dc52173032a856807a17ddef001"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.435702 14357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9d6c2e9a60f44aea9572361308f2cd2e"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "7088a95a46f018c9cba428adf6f51ed2"
server_key_iv: "23de1dc52173032a856807a17ddef001"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.436822 14357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:50:10.437479 14372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.437620 14357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:10.437693 14357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "9d6c2e9a60f44aea9572361308f2cd2e"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "7088a95a46f018c9cba428adf6f51ed2"
server_key_iv: "23de1dc52173032a856807a17ddef001"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.437757 14357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:10.444645 14357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:10.446413 14357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:10.446535 14357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:10.446722 14357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:10.447039 14357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:10.447093 14357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.447115 14357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:10.447137 14357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.452683 14357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33281
I20260927 02:50:10.452740 14485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33281 every 8 connection(s)
I20260927 02:50:10.453037 14357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:50:10.457316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14357
I20260927 02:50:10.457417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:50:10.457597 392 external_mini_cluster.cc:1428] Setting key 5aa283706cda32e3e18e0287dcdf34f8
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:10.462260 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.453687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35067 (local address 127.0.98.62:34859)
0927 02:50:10.453747 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.453749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.453915 (+ 166us) server_negotiation.cc:407] Connection header received
0927 02:50:10.454381 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.454383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.454392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.454443 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:10.455384 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.455889 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.456858 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.457056 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.457117 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.459620 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:10.459622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:10.459623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:10.459631 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:10.460878 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.461310 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.461310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.461311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.461359 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.461621 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.461622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.461622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.461754 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:10.461808 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.462123 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.462190 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:50:10.462527 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.453650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:35067)
0927 02:50:10.453800 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.453840 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.454307 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.454505 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.454509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.455107 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:10.455302 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.455308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.455958 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.455958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.456749 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.456751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.457150 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.457151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.457298 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.457936 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:10.457946 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:10.459509 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:10.460957 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.460964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.460966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.461242 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.461410 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.461411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.461411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.461541 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.461816 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:10.461822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:10.461993 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.462110 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.462280 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260927 02:50:10.463043 14486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260927 02:50:10.463135 14486 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:10.463306 14486 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260927 02:50:10.463671 14161 ts_manager.cc:194] Registered new tserver with Master: 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
I20260927 02:50:10.464051 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35067
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:50:10.481608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:10.559657 14491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:10.559856 14491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:10.559887 14491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:10.561034 14491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:10.561076 14491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:10.561110 14491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:50:10.562527 14491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:10.562753 14491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:10.562978 14491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:10.567611 14497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.567672 14499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.567775 14491 server_base.cc:1034] running on GCE node
W20260927 02:50:10.567847 14496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.568100 14491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:10.568338 14491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:10.569475 14491 hybrid_clock.cc:648] HybridClock initialized: now 1790477410569462 us; error 28 us; skew 500 ppm
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:10.571772 14491 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:50:10.572192 14491 webserver.cc:460] Webserver started at http://127.0.98.3:42409/ using document root <none> and password file <none>
I20260927 02:50:10.572383 14491 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:10.572430 14491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:10.572552 14491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:10.573568 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0c9d8769a5345d1a71eac9d835c27e3"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "d90a9db02d16ea986466350185475877"
server_key_iv: "9d1935fb5e6aed9a69fd175923849d68"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.573886 14491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0c9d8769a5345d1a71eac9d835c27e3"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "d90a9db02d16ea986466350185475877"
server_key_iv: "9d1935fb5e6aed9a69fd175923849d68"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.575119 14491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:50:10.575752 14506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.575892 14491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:10.575981 14491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "e0c9d8769a5345d1a71eac9d835c27e3"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "d90a9db02d16ea986466350185475877"
server_key_iv: "9d1935fb5e6aed9a69fd175923849d68"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.576038 14491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:10.604063 14491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:10.605849 14491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:10.605979 14491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:10.606177 14491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:10.606416 14491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:10.606444 14491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.606463 14491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:10.606480 14491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.611619 14491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33335
I20260927 02:50:10.611678 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33335 every 8 connection(s)
I20260927 02:50:10.611974 14491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:50:10.616128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14491
I20260927 02:50:10.616226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:50:10.616393 392 external_mini_cluster.cc:1428] Setting key f320b79a073cc0b24e4c1f2baf6d725d
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:10.620656 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.612633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54531 (local address 127.0.98.62:34859)
0927 02:50:10.612715 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.612718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.612860 (+ 142us) server_negotiation.cc:407] Connection header received
0927 02:50:10.613347 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.613348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.613358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.613397 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:10.614357 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.614907 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.615907 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.616181 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.616248 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.618283 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:10.618284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:10.618285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:10.618291 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:10.619465 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.619826 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.619827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.619828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.619879 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.620083 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.620084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.620084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.620194 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:10.620248 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.620496 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.620569 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 02:50:10.620842 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.612596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:54531)
0927 02:50:10.612747 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.612783 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.613260 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.613524 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.613527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.614097 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:10.614287 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.614293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.615014 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.615015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.615836 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.615839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.616261 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.616262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.616382 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.616903 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:10.616912 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:10.618201 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:10.619531 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.619538 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.619540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.619761 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.619931 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.619931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.619932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.620031 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.620246 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:10.620252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:10.620382 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.620467 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.620613 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260927 02:50:10.621212 14620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260927 02:50:10.621301 14620 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:10.621464 14620 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260927 02:50:10.621770 14161 ts_manager.cc:194] Registered new tserver with Master: e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335)
I20260927 02:50:10.622107 14161 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54531
I20260927 02:50:10.627947 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:50:10.634476 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.629195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.0.98.62:34859)
0927 02:50:10.629242 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.629244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.629343 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:50:10.629405 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.629406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.629416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.629457 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:10.629993 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.630461 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.631202 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.631400 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.631452 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.632491 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:10.632492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:10.632492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:10.632499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:10.633470 (+ 971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.633762 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.633763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.633764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.633819 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.633968 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.633969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.633969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.634112 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:10.634199 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.634312 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.634386 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:50:10.635721 14161 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:50:10.637218 14161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:50:10.643193 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.639141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33335 (local address 127.0.0.1:36030)
0927 02:50:10.639301 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.639338 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.639397 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.639625 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.639628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.639629 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:10.639693 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.639695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.640944 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.640945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.642103 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.642105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.642900 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.642901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.642973 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.642988 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.643048 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.643101 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":23,"thread_start_us":61,"threads_started":1}
I20260927 02:50:10.643383 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.639150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33281 (local address 127.0.0.1:57526)
0927 02:50:10.639563 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.639575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.639632 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.639904 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.639904 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.639906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:10.639955 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.639958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.641134 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.641135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.642325 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.642328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.643052 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.643053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.643115 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.643119 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.643176 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.643216 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":21,"thread_start_us":49,"threads_started":1}
I20260927 02:50:10.643504 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.639153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36257 (local address 127.0.0.1:44986)
0927 02:50:10.639685 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.639695 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.639733 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.640187 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.640188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.640190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:10.640231 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.640232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.641265 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.641266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.642363 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.642365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.643295 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.643296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.643355 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.643358 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.643400 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.643433 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":22,"thread_start_us":94,"threads_started":1}
I20260927 02:50:10.643514 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.639317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.0.98.3:33335)
0927 02:50:10.639483 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.639488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.639498 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:50:10.639536 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.639538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.639553 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.639622 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:10.639747 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.640836 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.642174 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.642831 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.642900 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.642958 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.643394 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.643438 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260927 02:50:10.643527 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.639399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.98.2:33281)
0927 02:50:10.639557 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.639561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.639697 (+ 136us) server_negotiation.cc:407] Connection header received
0927 02:50:10.639742 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.639745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.639760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.639826 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:10.640028 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.641055 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.642421 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.642975 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.643038 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.643092 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.643190 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.643234 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260927 02:50:10.643955 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.639519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.98.1:36257)
0927 02:50:10.639825 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.639829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.639838 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:50:10.640094 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.640097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.640112 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.640177 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:10.640313 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.641168 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.642433 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.643141 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.643208 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.643262 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.643830 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.643876 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":81,"threads_started":1}
I20260927 02:50:10.644214 14420 tablet_service.cc:1467] Processing CreateTablet for tablet dc82ad7250ec4b228f413803131f808f (DEFAULT_TABLE table=test-table [id=08b2622d0e194c57b5bd1cc6673610b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:10.644210 14554 tablet_service.cc:1467] Processing CreateTablet for tablet dc82ad7250ec4b228f413803131f808f (DEFAULT_TABLE table=test-table [id=08b2622d0e194c57b5bd1cc6673610b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:10.644498 14420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82ad7250ec4b228f413803131f808f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:10.644534 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82ad7250ec4b228f413803131f808f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:10.644559 14286 tablet_service.cc:1467] Processing CreateTablet for tablet dc82ad7250ec4b228f413803131f808f (DEFAULT_TABLE table=test-table [id=08b2622d0e194c57b5bd1cc6673610b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:10.644825 14286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc82ad7250ec4b228f413803131f808f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:10.646993 14634 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Bootstrap starting.
I20260927 02:50:10.647534 14634 tablet_bootstrap.cc:654] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:10.647644 14635 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Bootstrap starting.
I20260927 02:50:10.648066 14634 log.cc:824] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:10.648124 14636 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap starting.
I20260927 02:50:10.648269 14635 tablet_bootstrap.cc:654] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:10.648663 14636 tablet_bootstrap.cc:654] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:10.648707 14635 log.cc:824] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:10.649081 14636 log.cc:824] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:10.649323 14634 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: No bootstrap required, opened a new log
I20260927 02:50:10.649380 14634 ts_tablet_manager.cc:1397] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:10.649386 14635 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: No bootstrap required, opened a new log
I20260927 02:50:10.649430 14635 ts_tablet_manager.cc:1397] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:10.649772 14636 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: No bootstrap required, opened a new log
I20260927 02:50:10.649822 14636 ts_tablet_manager.cc:1397] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:10.650871 14634 raft_consensus.cc:348] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.651042 14634 raft_consensus.cc:374] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:10.651120 14634 raft_consensus.cc:729] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6c2e9a60f44aea9572361308f2cd2e, State: Initialized, Role: FOLLOWER
I20260927 02:50:10.651247 14634 consensus_queue.cc:260] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.651321 14635 raft_consensus.cc:348] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.651476 14635 raft_consensus.cc:374] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:10.651497 14634 ts_tablet_manager.cc:1428] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:50:10.651505 14635 raft_consensus.cc:729] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c9d8769a5345d1a71eac9d835c27e3, State: Initialized, Role: FOLLOWER
I20260927 02:50:10.651527 14486 heartbeater.cc:503] Master 127.0.98.62:34859 was elected leader, sending a full tablet report...
I20260927 02:50:10.651614 14635 consensus_queue.cc:260] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.651683 14636 raft_consensus.cc:348] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.651829 14636 raft_consensus.cc:374] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:10.651862 14636 raft_consensus.cc:729] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1b7b98551e431383cc1c9696b9e389, State: Initialized, Role: FOLLOWER
I20260927 02:50:10.651877 14635 ts_tablet_manager.cc:1428] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 02:50:10.651995 14620 heartbeater.cc:503] Master 127.0.98.62:34859 was elected leader, sending a full tablet report...
I20260927 02:50:10.651978 14636 consensus_queue.cc:260] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.652485 14636 ts_tablet_manager.cc:1428] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:50:10.652601 14352 heartbeater.cc:503] Master 127.0.98.62:34859 was elected leader, sending a full tablet report...
I20260927 02:50:10.670684 14642 raft_consensus.cc:482] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:10.670794 14642 raft_consensus.cc:504] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.671147 14642 leader_election.cc:288] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335), 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
I20260927 02:50:10.674630 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.671356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37411 (local address 127.0.98.3:33335)
0927 02:50:10.671413 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.671415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.671424 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:50:10.671451 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.671452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.671463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.671518 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:10.671668 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.672408 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.673738 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.674302 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.674355 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.674365 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.674524 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.674561 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:50:10.674630 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.671350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46345 (local address 127.0.98.2:33281)
0927 02:50:10.671405 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.671407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.671606 (+ 199us) server_negotiation.cc:407] Connection header received
0927 02:50:10.671677 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.671678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.671691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.671734 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:10.671839 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.672495 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.673733 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.674290 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.674355 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.674365 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.674524 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.674561 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:50:10.674646 14354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.671295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33335 (local address 127.0.98.1:37411)
0927 02:50:10.671354 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.671365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.671401 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.671527 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.671528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.671529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:10.671600 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.671602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.672474 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.672475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.673653 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.673656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.674369 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.674369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.674467 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.674469 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.674514 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.674557 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
I20260927 02:50:10.674904 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.671296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33281 (local address 127.0.98.1:46345)
0927 02:50:10.671543 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.671552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.671604 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.671746 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.671747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.671748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:10.671791 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.671792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.672553 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.672554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.673671 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.673672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.674369 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.674369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.674461 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.674465 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.674514 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.674557 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":86,"thread_start_us":102,"threads_started":1}
I20260927 02:50:10.675033 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82ad7250ec4b228f413803131f808f" candidate_uuid: "6a1b7b98551e431383cc1c9696b9e389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" is_pre_election: true
I20260927 02:50:10.675199 14574 raft_consensus.cc:2454] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1b7b98551e431383cc1c9696b9e389 in term 0.
I20260927 02:50:10.675210 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82ad7250ec4b228f413803131f808f" candidate_uuid: "6a1b7b98551e431383cc1c9696b9e389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" is_pre_election: true
I20260927 02:50:10.675346 14440 raft_consensus.cc:2454] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a1b7b98551e431383cc1c9696b9e389 in term 0.
I20260927 02:50:10.675421 14241 leader_election.cc:302] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389, e0c9d8769a5345d1a71eac9d835c27e3; no voters:
I20260927 02:50:10.675555 14642 raft_consensus.cc:2781] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:50:10.675613 14642 raft_consensus.cc:482] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:50:10.675629 14642 raft_consensus.cc:3037] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:10.676297 14642 raft_consensus.cc:504] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.676414 14642 leader_election.cc:288] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 1 election: Requested vote from peers e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335), 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
I20260927 02:50:10.676559 14440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82ad7250ec4b228f413803131f808f" candidate_uuid: "6a1b7b98551e431383cc1c9696b9e389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6c2e9a60f44aea9572361308f2cd2e"
I20260927 02:50:10.676635 14440 raft_consensus.cc:3037] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:10.677371 14440 raft_consensus.cc:2454] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1b7b98551e431383cc1c9696b9e389 in term 1.
I20260927 02:50:10.677479 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc82ad7250ec4b228f413803131f808f" candidate_uuid: "6a1b7b98551e431383cc1c9696b9e389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0c9d8769a5345d1a71eac9d835c27e3"
I20260927 02:50:10.677522 14239 leader_election.cc:302] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389, 9d6c2e9a60f44aea9572361308f2cd2e; no voters:
I20260927 02:50:10.677559 14574 raft_consensus.cc:3037] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:10.677608 14642 raft_consensus.cc:2781] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:10.677757 14642 raft_consensus.cc:686] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1b7b98551e431383cc1c9696b9e389, State: Running, Role: LEADER
I20260927 02:50:10.677850 14642 consensus_queue.cc:237] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.678386 14574 raft_consensus.cc:2454] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a1b7b98551e431383cc1c9696b9e389 in term 1.
I20260927 02:50:10.678669 14161 catalog_manager.cc:5495] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6a1b7b98551e431383cc1c9696b9e389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } health_report { overall_health: UNKNOWN } } }
I20260927 02:50:10.691780 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.689684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.0.98.1:36257)
0927 02:50:10.689727 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.689729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.689737 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:10.689769 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.689770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.689787 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.689842 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:50:10.689928 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.690391 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.691192 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.691369 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.691414 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.691511 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:50:10.691604 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:50:10.691657 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.691689 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.691705 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:50:10.693503 14574 raft_consensus.cc:1261] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 FOLLOWER]: Refusing update from remote peer 6a1b7b98551e431383cc1c9696b9e389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:10.693493 14440 raft_consensus.cc:1261] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Refusing update from remote peer 6a1b7b98551e431383cc1c9696b9e389: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:10.693848 14644 consensus_queue.cc:1035] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:10.693965 14642 consensus_queue.cc:1035] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:10.696592 14648 mvcc.cc:205] Tried to move back new op lower bound from 7333795474197876736 to 7333795474137067520. Current Snapshot: MvccSnapshot[applied={T|T < 7333795474197876736}]
I20260927 02:50:10.696807 14647 mvcc.cc:205] Tried to move back new op lower bound from 7333795474197876736 to 7333795474137067520. Current Snapshot: MvccSnapshot[applied={T|T < 7333795474197876736 or (T in {7333795474197876736})}]
I20260927 02:50:10.698303 14161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:10.698385 14161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:10.699131 14161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Sending DeleteTablet for 3 replicas of tablet dc82ad7250ec4b228f413803131f808f
I20260927 02:50:10.699412 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet dc82ad7250ec4b228f413803131f808f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36030
I20260927 02:50:10.699427 14420 tablet_service.cc:1514] Processing DeleteTablet for tablet dc82ad7250ec4b228f413803131f808f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57526
I20260927 02:50:10.699417 14286 tablet_service.cc:1514] Processing DeleteTablet for tablet dc82ad7250ec4b228f413803131f808f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44986
I20260927 02:50:10.699685 14656 tablet_replica.cc:330] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: stopping tablet replica
I20260927 02:50:10.699752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14223
I20260927 02:50:10.700368 14658 tablet_replica.cc:330] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: stopping tablet replica
I20260927 02:50:10.700460 14658 raft_consensus.cc:2229] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:10.700568 14658 raft_consensus.cc:2258] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:10.699784 14656 raft_consensus.cc:2229] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:10.700726 14656 raft_consensus.cc:2258] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:10.701159 14658 ts_tablet_manager.cc:1905] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:10.701207 14656 ts_tablet_manager.cc:1905] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:10.702625 14658 ts_tablet_manager.cc:1918] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:10.702694 14658 log.cc:1196] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/dc82ad7250ec4b228f413803131f808f
I20260927 02:50:10.702917 14656 ts_tablet_manager.cc:1918] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:10.702986 14656 log.cc:1196] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/dc82ad7250ec4b228f413803131f808f
I20260927 02:50:10.702984 14658 ts_tablet_manager.cc:1939] T dc82ad7250ec4b228f413803131f808f P 9d6c2e9a60f44aea9572361308f2cd2e: Deleting consensus metadata
I20260927 02:50:10.703219 14656 ts_tablet_manager.cc:1939] T dc82ad7250ec4b228f413803131f808f P e0c9d8769a5345d1a71eac9d835c27e3: Deleting consensus metadata
I20260927 02:50:10.703397 14145 catalog_manager.cc:4841] TS 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281): tablet dc82ad7250ec4b228f413803131f808f (table test-table [id=08b2622d0e194c57b5bd1cc6673610b2]) successfully deleted
I20260927 02:50:10.703706 14147 catalog_manager.cc:4841] TS e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335): tablet dc82ad7250ec4b228f413803131f808f (table test-table [id=08b2622d0e194c57b5bd1cc6673610b2]) successfully deleted
W20260927 02:50:10.705157 14146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36257 (error 108)
I20260927 02:50:10.705379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14357
I20260927 02:50:10.705893 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.705436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36257 (local address 127.0.0.1:45004)
0927 02:50:10.705827 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.705861 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372}
W20260927 02:50:10.706629 14146 catalog_manager.cc:4584] TS 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1:36257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc82ad7250ec4b228f413803131f808f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36257: connect: Connection refused (error 111)
I20260927 02:50:10.709486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14491
I20260927 02:50:10.713464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14129
I20260927 02:50:10.718067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--webserver_interface=127.0.98.62
--webserver_port=33903
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:10.796051 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:10.796252 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:10.796272 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:10.797360 14660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:10.797394 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:10.797439 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:10.797454 14660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:10.797470 14660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:10.798925 14660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:10.799115 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:10.799284 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:10.807142 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.807291 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.807142 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.807353 14660 server_base.cc:1034] running on GCE node
I20260927 02:50:10.807658 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:10.807976 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:10.809119 14660 hybrid_clock.cc:648] HybridClock initialized: now 1790477410809107 us; error 28 us; skew 500 ppm
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:10.811426 14660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:10.811890 14660 webserver.cc:460] Webserver started at http://127.0.98.62:33903/ using document root <none> and password file <none>
I20260927 02:50:10.812098 14660 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:10.812160 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:10.813462 14660 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:10.814045 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.814184 14660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:10.814268 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "c9fc18798e07479687455e9137520fdf"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "189504672861f1ca5339d44a36d151db"
server_key_iv: "b3856126ebaec6cfd5f8d32fb0574a64"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.814666 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:10.828022 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:10.829810 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:10.829937 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:10.833776 14660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34859
I20260927 02:50:10.833808 14728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34859 every 8 connection(s)
I20260927 02:50:10.834168 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:10.835824 14729 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:50:10.836524 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Bootstrap starting.
I20260927 02:50:10.842499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14660
I20260927 02:50:10.842862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36257
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34859
--webserver_port=42339
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:10.844772 14729 log.cc:824] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:10.846830 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:50:10.847092 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Bootstrap complete.
I20260927 02:50:10.849201 14729 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.849424 14729 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fc18798e07479687455e9137520fdf, State: Initialized, Role: FOLLOWER
I20260927 02:50:10.849565 14729 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.849630 14729 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:10.849663 14729 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:10.849707 14729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:50:10.850698 14729 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.850781 14729 leader_election.cc:302] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9fc18798e07479687455e9137520fdf; no voters:
I20260927 02:50:10.850950 14729 leader_election.cc:288] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:50:10.850996 14733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:50:10.851105 14733 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 LEADER]: Becoming Leader. State: Replica: c9fc18798e07479687455e9137520fdf, State: Running, Role: LEADER
I20260927 02:50:10.851186 14733 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:10.851212 14729 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:10.851728 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9fc18798e07479687455e9137520fdf. Latest consensus state: current_term: 2 leader_uuid: "c9fc18798e07479687455e9137520fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260927 02:50:10.851804 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:10.852062 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9fc18798e07479687455e9137520fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260927 02:50:10.852106 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:10.852116 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:10.852851 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:10.852953 14742 catalog_manager.cc:1259] Loaded cluster ID: 9ff8d8a352f04f9b8941d61fd1633218
I20260927 02:50:10.852988 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:50:10.853437 14742 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 02:50:10.858009 14742 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 02:50:10.860409 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:10.860623 14742 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Loaded TSK: 0
I20260927 02:50:10.860857 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:50:10.923895 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:10.924098 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:10.924129 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:10.925304 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:10.925349 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:10.925384 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:50:10.926797 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42339
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:10.927044 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:10.927237 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:10.932900 14756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.933012 14759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:10.932912 14757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:10.933116 14731 server_base.cc:1034] running on GCE node
I20260927 02:50:10.933339 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:10.933622 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:10.934800 14731 hybrid_clock.cc:648] HybridClock initialized: now 1790477410934785 us; error 26 us; skew 500 ppm
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:10.937729 14731 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:50:10.938288 14731 webserver.cc:460] Webserver started at http://127.0.98.1:42339/ using document root <none> and password file <none>
I20260927 02:50:10.938534 14731 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:10.938616 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:10.940320 14731 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:10.941026 14766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:10.941190 14731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:10.941269 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "6a1b7b98551e431383cc1c9696b9e389"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "6bc4ad83d4030dab1747cbb088690425"
server_key_iv: "4b16787167e5feec40f58ac25e3aa3ee"
server_key_version: "encryptionkey@0"
I20260927 02:50:10.941587 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:10.948508 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:10.950441 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:10.950548 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:10.950754 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:10.951187 14773 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:50:10.953084 14731 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:50:10.953137 14731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:50:10.953171 14731 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:50:10.953786 14731 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:50:10.953819 14731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:50:10.953896 14773 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap starting.
I20260927 02:50:10.960306 14731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36257
I20260927 02:50:10.960668 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:50:10.963068 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36257 every 8 connection(s)
I20260927 02:50:10.964308 14773 log.cc:824] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:10.967065 14773 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:50:10.967356 14773 tablet_bootstrap.cc:492] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap complete.
I20260927 02:50:10.967911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14731
I20260927 02:50:10.968194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33281
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34859
--webserver_port=34125
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:10.968437 14773 ts_tablet_manager.cc:1397] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
Sep 27 02:50:10 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:10.969714 14773 raft_consensus.cc:348] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.970078 14773 raft_consensus.cc:729] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1b7b98551e431383cc1c9696b9e389, State: Initialized, Role: FOLLOWER
I20260927 02:50:10.970222 14773 consensus_queue.cc:260] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } }
I20260927 02:50:10.970500 14773 ts_tablet_manager.cc:1428] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260927 02:50:10.973781 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.961431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41777 (local address 127.0.98.62:34859)
0927 02:50:10.963991 (+ 2560us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.963998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.964024 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:50:10.964124 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.964128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.964168 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.964287 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:10.965054 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.965951 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.966994 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.967431 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.967563 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.970083 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:10.970088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:10.970088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:10.970100 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:10.971821 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.972266 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.972269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.972270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.972377 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:10.972648 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:10.972649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:10.972649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:10.972929 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:10.972987 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.973406 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.973564 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2508,"thread_start_us":69,"threads_started":1}
I20260927 02:50:10.974368 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.961351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:41777)
0927 02:50:10.963321 (+ 1970us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.963360 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.963824 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.964336 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.964340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.964699 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:10.964928 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.964934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.966070 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.966071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.966911 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.966914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.967552 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.967554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.967682 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.968317 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:10.968328 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:10.969951 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:10.971905 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.971911 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.971913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.972177 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.972442 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:10.972443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:10.972443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:10.972581 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:10.973035 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:10.973042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:10.973202 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.973989 (+ 787us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.974174 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"thread_start_us":97,"threads_started":1}
I20260927 02:50:10.975281 14881 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260927 02:50:10.975375 14881 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:10.975544 14881 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260927 02:50:10.976305 14693 ts_manager.cc:194] Registered new tserver with Master: 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1:36257)
I20260927 02:50:10.977478 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41777
I20260927 02:50:10.978458 14881 heartbeater.cc:503] Master 127.0.98.62:34859 was elected leader, sending a full tablet report...
I20260927 02:50:10.983062 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.976908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36257 (local address 127.0.0.1:45010)
0927 02:50:10.978838 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:10.978873 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:10.978937 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:10.979237 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:10.979239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:10.979242 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:10.979305 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.979307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.980559 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.980561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.981754 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:10.981757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.982723 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.982724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:10.982842 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.982858 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:50:10.982921 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:50:10.982976 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":92,"threads_started":1}
I20260927 02:50:10.983425 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:10.977775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.0.98.1:36257)
0927 02:50:10.978668 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:10.978674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:10.979011 (+ 337us) server_negotiation.cc:407] Connection header received
0927 02:50:10.979122 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:10.979126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:10.979142 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:10.979228 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:10.979367 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.980460 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.981835 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:10.982561 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:10.982630 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:10.982688 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:10.983317 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 02:50:10.983361 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":82,"threads_started":1}
I20260927 02:50:10.983919 14797 tablet_service.cc:1514] Processing DeleteTablet for tablet dc82ad7250ec4b228f413803131f808f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45010
I20260927 02:50:10.984516 14890 tablet_replica.cc:330] stopping tablet replica
I20260927 02:50:10.984624 14890 raft_consensus.cc:2229] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:10.984687 14890 raft_consensus.cc:2258] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:10.984972 14890 ts_tablet_manager.cc:1905] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:10.987159 14890 ts_tablet_manager.cc:1918] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:10.987229 14890 log.cc:1196] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/dc82ad7250ec4b228f413803131f808f
I20260927 02:50:10.987438 14890 ts_tablet_manager.cc:1939] T dc82ad7250ec4b228f413803131f808f P 6a1b7b98551e431383cc1c9696b9e389: Deleting consensus metadata
I20260927 02:50:10.988006 14678 catalog_manager.cc:4841] TS 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1:36257): tablet dc82ad7250ec4b228f413803131f808f (table test-table [id=08b2622d0e194c57b5bd1cc6673610b2]) successfully deleted
W20260927 02:50:11.052738 14886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:11.052946 14886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:11.052977 14886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:11.054112 14886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:11.054157 14886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:11.054190 14886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:50:11.055737 14886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34125
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:11.055982 14886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:11.056193 14886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:11.060858 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:11.060931 14898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:11.060853 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:11.061194 14886 server_base.cc:1034] running on GCE node
I20260927 02:50:11.061363 14886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:11.061580 14886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:11.062714 14886 hybrid_clock.cc:648] HybridClock initialized: now 1790477411062699 us; error 30 us; skew 500 ppm
Sep 27 02:50:11 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:11.064956 14886 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:50:11.065382 14886 webserver.cc:460] Webserver started at http://127.0.98.2:34125/ using document root <none> and password file <none>
I20260927 02:50:11.065550 14886 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:11.065599 14886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:11.066892 14886 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:11.067447 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.067651 14886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:11.067739 14886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "9d6c2e9a60f44aea9572361308f2cd2e"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "7088a95a46f018c9cba428adf6f51ed2"
server_key_iv: "23de1dc52173032a856807a17ddef001"
server_key_version: "encryptionkey@0"
I20260927 02:50:11.068084 14886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:11.087364 14886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:11.089243 14886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:11.089367 14886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:11.089592 14886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:11.089895 14886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:11.089929 14886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.089959 14886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:11.089977 14886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.095453 14886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33281
I20260927 02:50:11.095510 15018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33281 every 8 connection(s)
I20260927 02:50:11.095837 14886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:50:11 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:11.104620 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.096486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42073 (local address 127.0.98.62:34859)
0927 02:50:11.096557 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.096559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.096892 (+ 333us) server_negotiation.cc:407] Connection header received
0927 02:50:11.097370 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.097371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.097386 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.097437 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:11.098376 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.098904 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.099842 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.100046 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.100099 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.102071 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:11.102072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:11.102073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:11.102082 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:11.103379 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:11.103751 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:11.103752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:11.103752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:11.103803 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:11.104016 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:11.104017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:11.104017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:11.104122 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:11.104176 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.104421 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.104515 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:50:11.104768 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.096467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:42073)
0927 02:50:11.096752 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.096789 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:11.097275 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:11.097533 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:11.097537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:11.098088 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:11.098282 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.098286 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.098978 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.098979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.099766 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.099768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.100110 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.100111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.100233 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.100704 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:11.100712 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:11.101985 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:11.103452 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:11.103459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:11.103461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:11.103685 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:11.103857 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:11.103858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:11.103858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:11.103961 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:11.104174 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:11.104180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:11.104310 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 02:50:11.104396 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:50:11.104549 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":79,"threads_started":1}
I20260927 02:50:11.105165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 14886
I20260927 02:50:11.105211 15019 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260927 02:50:11.105296 15019 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:11.105434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33335
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34859
--webserver_port=42409
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:11.105479 15019 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260927 02:50:11.105878 14693 ts_manager.cc:194] Registered new tserver with Master: 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
I20260927 02:50:11.106261 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42073
W20260927 02:50:11.185204 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:11.185420 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:11.185444 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:11.186554 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:11.186587 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:11.186612 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:50:11.188087 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42409
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:11.188283 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:11.188463 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:11.193391 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:11.193418 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:11.193408 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:11.193601 15022 server_base.cc:1034] running on GCE node
I20260927 02:50:11.193816 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:11.194070 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:11.195212 15022 hybrid_clock.cc:648] HybridClock initialized: now 1790477411195202 us; error 31 us; skew 500 ppm
Sep 27 02:50:11 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:11.197602 15022 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:50:11.198035 15022 webserver.cc:460] Webserver started at http://127.0.98.3:42409/ using document root <none> and password file <none>
I20260927 02:50:11.198240 15022 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:11.198302 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:11.199625 15022 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:11.200187 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.200332 15022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:11.200410 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "e0c9d8769a5345d1a71eac9d835c27e3"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "d90a9db02d16ea986466350185475877"
server_key_iv: "9d1935fb5e6aed9a69fd175923849d68"
server_key_version: "encryptionkey@0"
I20260927 02:50:11.200760 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:11.219569 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:11.221326 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:11.221457 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:11.221664 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:11.221956 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:11.221987 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.222015 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:11.222038 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.227308 15022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33335
I20260927 02:50:11.227370 15150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33335 every 8 connection(s)
I20260927 02:50:11.227689 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:50:11.230154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15022
Sep 27 02:50:11 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:11.236601 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.230473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40846 (local address 127.0.98.62:34859)
0927 02:50:11.230584 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.230586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.230594 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:11.230670 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.230671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.230681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.230729 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:11.231273 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.231746 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.232546 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.232745 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.232795 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.233978 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:11.233979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:11.233980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:11.233985 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:11.235543 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:11.235908 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:11.235908 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:11.235909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:11.235956 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:11.236133 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:11.236134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:11.236134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:11.236233 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:11.236397 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.236432 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.236508 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260927 02:50:11.237097 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.228329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:38339)
0927 02:50:11.228464 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.228499 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:11.228950 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:11.229172 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:11.229176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:11.229752 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:11.229940 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.229945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.230694 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.230695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.231524 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.231527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.231866 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.231867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.231984 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.232452 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:11.232459 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:11.233990 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:11.235676 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:11.235684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:11.235686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:11.235957 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:11.236111 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:11.236112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:11.236112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:11.236231 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:11.236457 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:11.236463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:11.236619 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 02:50:11.236715 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 02:50:11.236871 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260927 02:50:11.237399 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.228308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38339 (local address 127.0.98.62:34859)
0927 02:50:11.228369 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.228371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.228571 (+ 200us) server_negotiation.cc:407] Connection header received
0927 02:50:11.229025 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.229026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.229039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.229101 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:11.230014 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.230608 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.231603 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.231803 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.231855 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.234064 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:11.234064 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:11.234065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:11.234070 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:11.235613 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:11.236014 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:11.236014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:11.236015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:11.236058 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:11.236287 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:11.236288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:11.236288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:11.236390 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:11.236448 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.237247 (+ 799us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.237323 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:50:11.237709 15151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260927 02:50:11.237803 15151 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:11.237969 15151 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260927 02:50:11.238292 14693 ts_manager.cc:194] Registered new tserver with Master: e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335)
I20260927 02:50:11.238628 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38339
I20260927 02:50:11.248518 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:50:11.251259 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.248848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.98.62:34859)
0927 02:50:11.248900 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.248902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.249039 (+ 137us) server_negotiation.cc:407] Connection header received
0927 02:50:11.249112 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.249113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.249150 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.249211 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:50:11.249324 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.249803 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.250512 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.250731 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.250788 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.250963 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:50:11.251077 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:50:11.251148 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.251168 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.251186 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:50:11.251514 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:50:11.253101 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:50:11.255046 14797 tablet_service.cc:1467] Processing CreateTablet for tablet 99de37630f6e43db98d845311cbf6b09 (DEFAULT_TABLE table=test-table [id=d3f6e7ecc48044e0b8cb5902fa02b8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:11.255239 14797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99de37630f6e43db98d845311cbf6b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:11.256534 14773 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap starting.
I20260927 02:50:11.256907 14773 tablet_bootstrap.cc:654] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:11.257474 14773 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: No bootstrap required, opened a new log
I20260927 02:50:11.257527 14773 ts_tablet_manager.cc:1397] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 02:50:11.257632 14773 raft_consensus.cc:348] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.257714 14773 raft_consensus.cc:374] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:11.257764 14773 raft_consensus.cc:729] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1b7b98551e431383cc1c9696b9e389, State: Initialized, Role: FOLLOWER
I20260927 02:50:11.257803 14773 consensus_queue.cc:260] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.257891 14773 ts_tablet_manager.cc:1428] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.258880 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.254837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33335 (local address 127.0.0.1:36042)
0927 02:50:11.255088 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.255098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:11.255210 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:11.255360 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:11.255361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:11.255362 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:11.255408 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.255409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.256368 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.256369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.257418 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.257420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.258610 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.258611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.258676 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.258680 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:11.258734 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:50:11.258772 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":111,"thread_start_us":84,"threads_started":1}
I20260927 02:50:11.258888 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.255001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.0.98.3:33335)
0927 02:50:11.255183 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.255188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.255199 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:50:11.255279 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.255281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.255296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.255365 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:11.255457 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.256284 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.257604 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.258541 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.258594 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.258636 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.258748 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.258776 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260927 02:50:11.259575 15085 tablet_service.cc:1467] Processing CreateTablet for tablet 99de37630f6e43db98d845311cbf6b09 (DEFAULT_TABLE table=test-table [id=d3f6e7ecc48044e0b8cb5902fa02b8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:11.259756 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.254967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.98.2:33281)
0927 02:50:11.256615 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.256619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.256629 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:50:11.256669 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.256671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.256686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.256761 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:11.256854 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.257812 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.258979 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.259456 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.259512 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.259549 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.259673 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.259700 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1603,"thread_start_us":86,"threads_started":1}
I20260927 02:50:11.259761 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.254836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33281 (local address 127.0.0.1:57542)
0927 02:50:11.254874 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.254884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:11.254916 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:11.256756 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:11.256757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:11.256759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:11.256797 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.256798 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.257920 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.257921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.258897 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.258899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.259534 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.259535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.259597 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.259601 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:11.259662 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:50:11.259703 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:50:11.259936 15085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99de37630f6e43db98d845311cbf6b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:11.260377 14953 tablet_service.cc:1467] Processing CreateTablet for tablet 99de37630f6e43db98d845311cbf6b09 (DEFAULT_TABLE table=test-table [id=d3f6e7ecc48044e0b8cb5902fa02b8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:11.260607 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99de37630f6e43db98d845311cbf6b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:11.262429 15161 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Bootstrap starting.
I20260927 02:50:11.263010 15162 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Bootstrap starting.
I20260927 02:50:11.263119 15161 tablet_bootstrap.cc:654] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:11.263551 15161 log.cc:824] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:11.263561 15162 tablet_bootstrap.cc:654] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:11.263973 15162 log.cc:824] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:11.264179 15161 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: No bootstrap required, opened a new log
I20260927 02:50:11.264250 15161 ts_tablet_manager.cc:1397] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:11.264593 15162 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: No bootstrap required, opened a new log
I20260927 02:50:11.264657 15162 ts_tablet_manager.cc:1397] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:11.265514 15161 raft_consensus.cc:348] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.265656 15161 raft_consensus.cc:374] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:11.265688 15161 raft_consensus.cc:729] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0c9d8769a5345d1a71eac9d835c27e3, State: Initialized, Role: FOLLOWER
I20260927 02:50:11.265800 15161 consensus_queue.cc:260] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.266038 15161 ts_tablet_manager.cc:1428] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:50:11.266084 15151 heartbeater.cc:503] Master 127.0.98.62:34859 was elected leader, sending a full tablet report...
I20260927 02:50:11.265980 15162 raft_consensus.cc:348] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.266135 15162 raft_consensus.cc:374] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:11.266165 15162 raft_consensus.cc:729] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6c2e9a60f44aea9572361308f2cd2e, State: Initialized, Role: FOLLOWER
I20260927 02:50:11.266265 15162 consensus_queue.cc:260] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.266487 15162 ts_tablet_manager.cc:1428] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:50:11.266602 15019 heartbeater.cc:503] Master 127.0.98.62:34859 was elected leader, sending a full tablet report...
W20260927 02:50:11.346338 15020 tablet.cc:2369] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:50:11.463647 14882 tablet.cc:2369] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:50:11.478259 15152 tablet.cc:2369] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:50:11.554970 15165 raft_consensus.cc:482] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:11.555105 15165 raft_consensus.cc:504] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.555487 15165 leader_election.cc:288] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281), 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1:36257)
I20260927 02:50:11.558231 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.555773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34573 (local address 127.0.98.1:36257)
0927 02:50:11.555890 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.555893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.555899 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:50:11.555922 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.555923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.555931 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.555984 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:11.556132 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.556609 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.557534 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.557915 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.557975 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.557987 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.558122 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.558159 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260927 02:50:11.558251 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.555655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36257 (local address 127.0.98.3:34573)
0927 02:50:11.555755 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.555766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:11.555811 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:11.556004 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:11.556006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:11.556007 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:11.556070 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.556071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.556684 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.556685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.557446 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.557449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.557986 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.557987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.558063 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.558066 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:50:11.558110 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:50:11.558149 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":25}
I20260927 02:50:11.558249 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.555782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42021 (local address 127.0.98.2:33281)
0927 02:50:11.555833 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.555835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.555947 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:50:11.556002 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.556003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.556015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.556055 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:11.556205 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.556650 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.557548 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.557945 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.558004 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.558014 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.558152 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.558186 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:50:11.558264 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.555676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33281 (local address 127.0.98.3:42021)
0927 02:50:11.555879 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.555886 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:11.555942 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:11.556065 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:11.556066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:11.556067 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:11.556144 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.556146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.556717 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.556718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.557465 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:11.557467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.558020 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.558021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:11.558084 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.558087 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:50:11.558132 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:50:11.558172 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260927 02:50:11.558543 14835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99de37630f6e43db98d845311cbf6b09" candidate_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1b7b98551e431383cc1c9696b9e389" is_pre_election: true
I20260927 02:50:11.558717 14835 raft_consensus.cc:2454] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0c9d8769a5345d1a71eac9d835c27e3 in term 0.
I20260927 02:50:11.558754 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99de37630f6e43db98d845311cbf6b09" candidate_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" is_pre_election: true
I20260927 02:50:11.558919 14973 raft_consensus.cc:2454] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0c9d8769a5345d1a71eac9d835c27e3 in term 0.
I20260927 02:50:11.558952 15039 leader_election.cc:302] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389, e0c9d8769a5345d1a71eac9d835c27e3; no voters:
I20260927 02:50:11.559101 15165 raft_consensus.cc:2781] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:50:11.559160 15165 raft_consensus.cc:482] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:50:11.559206 15165 raft_consensus.cc:3037] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:11.559885 15165 raft_consensus.cc:504] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.560003 15165 leader_election.cc:288] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [CANDIDATE]: Term 1 election: Requested vote from peers 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281), 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1:36257)
I20260927 02:50:11.560144 14835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99de37630f6e43db98d845311cbf6b09" candidate_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a1b7b98551e431383cc1c9696b9e389"
I20260927 02:50:11.560182 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99de37630f6e43db98d845311cbf6b09" candidate_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6c2e9a60f44aea9572361308f2cd2e"
I20260927 02:50:11.560215 14835 raft_consensus.cc:3037] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:11.560252 14973 raft_consensus.cc:3037] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:11.560814 14835 raft_consensus.cc:2454] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0c9d8769a5345d1a71eac9d835c27e3 in term 1.
I20260927 02:50:11.560956 15039 leader_election.cc:302] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389, e0c9d8769a5345d1a71eac9d835c27e3; no voters:
I20260927 02:50:11.561045 15165 raft_consensus.cc:2781] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:11.561048 14973 raft_consensus.cc:2454] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0c9d8769a5345d1a71eac9d835c27e3 in term 1.
I20260927 02:50:11.561190 15165 raft_consensus.cc:686] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 LEADER]: Becoming Leader. State: Replica: e0c9d8769a5345d1a71eac9d835c27e3, State: Running, Role: LEADER
I20260927 02:50:11.561283 15165 consensus_queue.cc:237] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.562021 14693 catalog_manager.cc:5495] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } health_report { overall_health: UNKNOWN } } }
I20260927 02:50:11.609333 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.607279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.0.98.3:33335)
0927 02:50:11.607327 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:11.607329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:11.607337 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:11.607377 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:11.607377 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:11.607396 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:11.607459 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:50:11.607600 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.608048 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.608741 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:11.608933 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:11.608976 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:11.609090 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:50:11.609191 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:50:11.609244 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:11.609256 (+ 12us) server_negotiation.cc:299] Negotiation successful
0927 02:50:11.609273 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:50:11.610989 14835 raft_consensus.cc:1261] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Refusing update from remote peer e0c9d8769a5345d1a71eac9d835c27e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:11.610991 14973 raft_consensus.cc:1261] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Refusing update from remote peer e0c9d8769a5345d1a71eac9d835c27e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:11.611316 15169 consensus_queue.cc:1035] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:11.611441 15165 consensus_queue.cc:1035] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:11.613458 15173 mvcc.cc:205] Tried to move back new op lower bound from 7333795477956128768 to 7333795477755551744. Current Snapshot: MvccSnapshot[applied={T|T < 7333795477956128768}]
I20260927 02:50:11.615160 14692 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:11.615249 14692 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:11.615952 14692 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Sending DeleteTablet for 3 replicas of tablet 99de37630f6e43db98d845311cbf6b09
I20260927 02:50:11.616219 14953 tablet_service.cc:1514] Processing DeleteTablet for tablet 99de37630f6e43db98d845311cbf6b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57542
I20260927 02:50:11.616214 15085 tablet_service.cc:1514] Processing DeleteTablet for tablet 99de37630f6e43db98d845311cbf6b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36042
I20260927 02:50:11.616492 14797 tablet_service.cc:1514] Processing DeleteTablet for tablet 99de37630f6e43db98d845311cbf6b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45010
I20260927 02:50:11.616508 15182 tablet_replica.cc:330] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: stopping tablet replica
I20260927 02:50:11.616598 15182 raft_consensus.cc:2229] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:11.616706 15183 tablet_replica.cc:330] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: stopping tablet replica
I20260927 02:50:11.616703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14731
I20260927 02:50:11.616724 15182 raft_consensus.cc:2258] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:11.616813 15183 raft_consensus.cc:2229] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:11.616899 15183 raft_consensus.cc:2258] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:11.617125 15182 ts_tablet_manager.cc:1905] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:11.618363 15181 tablet_replica.cc:330] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: stopping tablet replica
I20260927 02:50:11.618455 15181 raft_consensus.cc:2229] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:50:11.618543 15181 raft_consensus.cc:2258] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:11.618669 15182 ts_tablet_manager.cc:1918] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:11.618772 15182 log.cc:1196] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/99de37630f6e43db98d845311cbf6b09
I20260927 02:50:11.618988 15181 ts_tablet_manager.cc:1905] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:11.619158 15182 ts_tablet_manager.cc:1939] T 99de37630f6e43db98d845311cbf6b09 P 9d6c2e9a60f44aea9572361308f2cd2e: Deleting consensus metadata
I20260927 02:50:11.619570 14677 catalog_manager.cc:4841] TS 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281): tablet 99de37630f6e43db98d845311cbf6b09 (table test-table [id=d3f6e7ecc48044e0b8cb5902fa02b8e9]) successfully deleted
I20260927 02:50:11.620594 15181 ts_tablet_manager.cc:1918] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:11.620652 15181 log.cc:1196] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/99de37630f6e43db98d845311cbf6b09
I20260927 02:50:11.620898 15181 ts_tablet_manager.cc:1939] T 99de37630f6e43db98d845311cbf6b09 P e0c9d8769a5345d1a71eac9d835c27e3: Deleting consensus metadata
I20260927 02:50:11.621217 14679 catalog_manager.cc:4841] TS e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335): tablet 99de37630f6e43db98d845311cbf6b09 (table test-table [id=d3f6e7ecc48044e0b8cb5902fa02b8e9]) successfully deleted
W20260927 02:50:11.623664 14678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36257 (error 108)
I20260927 02:50:11.623922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14886
I20260927 02:50:11.624079 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:11.623985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36257 (local address 127.0.0.1:45016)
0927 02:50:11.624026 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.624052 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:50:11.624730 14678 catalog_manager.cc:4584] TS 6a1b7b98551e431383cc1c9696b9e389 (127.0.98.1:36257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99de37630f6e43db98d845311cbf6b09: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36257: connect: Connection refused (error 111)
I20260927 02:50:11.628506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 15022
I20260927 02:50:11.632486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 14660
I20260927 02:50:11.636835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--webserver_interface=127.0.98.62
--webserver_port=33903
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:11.714865 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:11.715061 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:11.715081 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:11.716223 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:11.716262 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:11.716274 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:11.716285 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:11.716295 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:11.717730 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:11.717944 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:11.718153 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:11.722702 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:11.722870 15185 server_base.cc:1034] running on GCE node
W20260927 02:50:11.722715 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:11.722733 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:11.723155 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:11.723408 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:11.724534 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790477411724522 us; error 24 us; skew 500 ppm
Sep 27 02:50:11 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:11.730206 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:11.730597 15185 webserver.cc:460] Webserver started at http://127.0.98.62:33903/ using document root <none> and password file <none>
I20260927 02:50:11.730841 15185 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:11.730894 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:11.732103 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:50:11.732681 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.732815 15185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:11.732888 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "c9fc18798e07479687455e9137520fdf"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "189504672861f1ca5339d44a36d151db"
server_key_iv: "b3856126ebaec6cfd5f8d32fb0574a64"
server_key_version: "encryptionkey@0"
I20260927 02:50:11.733220 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:11.748736 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:11.750473 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:11.750578 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:11.754154 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34859
I20260927 02:50:11.754185 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34859 every 8 connection(s)
I20260927 02:50:11.754539 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:11.756184 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:50:11.756826 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Bootstrap starting.
I20260927 02:50:11.761085 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15185
I20260927 02:50:11.761312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36257
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34859
--webserver_port=42339
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:11.764899 15253 log.cc:824] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:11.767115 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:50:11.767354 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf: Bootstrap complete.
I20260927 02:50:11.769383 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:11.769611 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fc18798e07479687455e9137520fdf, State: Initialized, Role: FOLLOWER
I20260927 02:50:11.769769 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:11.769842 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:11.769891 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:11.769936 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:50:11.770924 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:11.771020 15253 leader_election.cc:302] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9fc18798e07479687455e9137520fdf; no voters:
I20260927 02:50:11.771188 15253 leader_election.cc:288] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:50:11.771234 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:50:11.771353 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [term 3 LEADER]: Becoming Leader. State: Replica: c9fc18798e07479687455e9137520fdf, State: Running, Role: LEADER
I20260927 02:50:11.771447 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:11.771435 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260927 02:50:11.771692 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9fc18798e07479687455e9137520fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260927 02:50:11.771781 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:11.771941 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9fc18798e07479687455e9137520fdf. Latest consensus state: current_term: 3 leader_uuid: "c9fc18798e07479687455e9137520fdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fc18798e07479687455e9137520fdf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260927 02:50:11.771993 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fc18798e07479687455e9137520fdf [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:11.771986 15262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:11.773113 15262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:11.773211 15262 catalog_manager.cc:1259] Loaded cluster ID: 9ff8d8a352f04f9b8941d61fd1633218
I20260927 02:50:11.773239 15262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 02:50:11.773528 15262 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe4becff9ad google::LogMessage::Fail() at ??:0
@ 0x7fe4bed0288a google::LogMessage::SendToLog() at ??:0
@ 0x7fe4becff4de google::LogMessage::Flush() at ??:0
@ 0x7fe4bed00a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4c6c240f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe4c6c2d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe4c0bcd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe4c0bc2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4c1a266db start_thread at ??:0
@ 0x7fe4bd7fd61f clone at ??:0
W20260927 02:50:11.870018 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:11.870222 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:11.870245 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:11.871415 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:11.871459 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:11.871484 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:50:11.872946 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42339
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:11.873179 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:11.873389 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:11.879151 15290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:11.879271 15255 server_base.cc:1034] running on GCE node
W20260927 02:50:11.879160 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:11.879350 15289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:11.879546 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:11.879828 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:11.881008 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790477411880996 us; error 27 us; skew 500 ppm
Sep 27 02:50:11 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:11.883908 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:50:11.884461 15255 webserver.cc:460] Webserver started at http://127.0.98.1:42339/ using document root <none> and password file <none>
I20260927 02:50:11.884742 15255 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:11.884807 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:11.886520 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:50:11.887229 15299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.887388 15255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:11.887492 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "6a1b7b98551e431383cc1c9696b9e389"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "6bc4ad83d4030dab1747cbb088690425"
server_key_iv: "4b16787167e5feec40f58ac25e3aa3ee"
server_key_version: "encryptionkey@0"
I20260927 02:50:11.887918 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:11.913839 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:11.916038 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:11.916195 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:11.916412 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:11.916857 15306 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:50:11.918390 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:50:11.918435 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:50:11.918466 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:50:11.919116 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:50:11.919152 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:50:11.919243 15306 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap starting.
I20260927 02:50:11.925215 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36257
I20260927 02:50:11.925293 15413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36257 every 8 connection(s)
I20260927 02:50:11.925649 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:50:11.928406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15255
I20260927 02:50:11.928686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33281
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34859
--webserver_port=34125
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:11.929553 15306 log.cc:824] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:11.933054 15306 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:50:11.933389 15306 tablet_bootstrap.cc:492] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Bootstrap complete.
I20260927 02:50:11.934511 15306 ts_tablet_manager.cc:1397] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260927 02:50:11.935998 15306 raft_consensus.cc:348] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.936344 15306 raft_consensus.cc:729] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1b7b98551e431383cc1c9696b9e389, State: Initialized, Role: FOLLOWER
I20260927 02:50:11.936487 15306 consensus_queue.cc:260] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:11.936789 15306 ts_tablet_manager.cc:1428] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260927 02:50:12.010787 15417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:12.011035 15417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:12.011059 15417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:12.012233 15417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:12.012276 15417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:12.012302 15417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:50:12.013800 15417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34125
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:12.014034 15417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:12.014259 15417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:12.019382 15425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:12.019395 15424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:12.019371 15427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:12.019728 15417 server_base.cc:1034] running on GCE node
I20260927 02:50:12.019939 15417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:12.020179 15417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260927 02:50:12.020586 15416 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:50:11.926377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:42697)
0927 02:50:11.926560 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:11.926612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:11.927278 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:12.020440 (+ 93162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
W20260927 02:50:12.020774 15414 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34859 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34859: Client connection negotiation failed: client connection to 127.0.98.62:34859: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:50:12.021056 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.020975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:43887)
0927 02:50:12.021007 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.021040 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 02:50:12.021317 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.021233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:58617)
0927 02:50:12.021272 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.021296 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:50:12.021327 15417 hybrid_clock.cc:648] HybridClock initialized: now 1790477412021298 us; error 39 us; skew 500 ppm
W20260927 02:50:12.021420 15414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 02:50:12 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:12.023808 15417 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:50:12.024202 15417 webserver.cc:460] Webserver started at http://127.0.98.2:34125/ using document root <none> and password file <none>
I20260927 02:50:12.024371 15417 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:12.024411 15417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:12.025705 15417 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:12.026247 15434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:12.026392 15417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:12.026473 15417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "9d6c2e9a60f44aea9572361308f2cd2e"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "7088a95a46f018c9cba428adf6f51ed2"
server_key_iv: "23de1dc52173032a856807a17ddef001"
server_key_version: "encryptionkey@0"
I20260927 02:50:12.026872 15417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:12.035634 15417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:12.037601 15417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:12.037720 15417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:12.037902 15417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:12.038144 15417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:12.038173 15417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:12.038193 15417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:12.038210 15417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:12.043819 15417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33281
I20260927 02:50:12.043869 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33281 every 8 connection(s)
I20260927 02:50:12.044191 15417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:50:12.044896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15417
I20260927 02:50:12.045183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33335
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34859
--webserver_port=42409
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:12.045282 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.044837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:35171)
0927 02:50:12.044998 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.045095 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
W20260927 02:50:12.045634 15548 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34859 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34859: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
I20260927 02:50:12.045950 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.045863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:41115)
0927 02:50:12.045906 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.045932 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:50:12.046190 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.046112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:33391)
0927 02:50:12.046159 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.046178 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 02:50:12.046288 15548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:50:12.125200 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:12.125404 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:12.125425 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:12.126606 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:12.126657 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:12.126684 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:50:12.128088 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42409
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:12.128321 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:12.128508 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:12.133364 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:12.133364 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:12.133389 15551 server_base.cc:1034] running on GCE node
W20260927 02:50:12.133414 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:12.133772 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:12.133991 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:12.135125 15551 hybrid_clock.cc:648] HybridClock initialized: now 1790477412135112 us; error 28 us; skew 500 ppm
Sep 27 02:50:12 dist-test-slave-f4rg krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:12.137486 15551 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:50:12.137864 15551 webserver.cc:460] Webserver started at http://127.0.98.3:42409/ using document root <none> and password file <none>
I20260927 02:50:12.138032 15551 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:12.138069 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:12.139314 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:50:12.139871 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:12.140005 15551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:12.140084 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "e0c9d8769a5345d1a71eac9d835c27e3"
format_stamp: "Formatted at 2026-09-27 02:50:10 on dist-test-slave-f4rg"
server_key: "d90a9db02d16ea986466350185475877"
server_key_iv: "9d1935fb5e6aed9a69fd175923849d68"
server_key_version: "encryptionkey@0"
I20260927 02:50:12.140398 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:12.150705 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:12.152803 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:12.152933 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:12.153146 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:12.153434 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:12.153465 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:12.153493 15551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:12.153517 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:12.158980 15551 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33335
I20260927 02:50:12.159032 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33335 every 8 connection(s)
I20260927 02:50:12.159360 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:50:12.160018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15551
I20260927 02:50:12.160542 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.160028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:49745)
0927 02:50:12.160249 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.160344 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
W20260927 02:50:12.160959 15680 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34859 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34859: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
I20260927 02:50:12.161227 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.161142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:38815)
0927 02:50:12.161185 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.161209 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:50:12.161463 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:12.161401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:32941)
0927 02:50:12.161432 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:12.161451 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:50:12.161558 15680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:50:13.021926 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:13.021724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:46975)
0927 02:50:13.021837 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:13.021867 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260927 02:50:13.046873 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:13.046636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:39739)
0927 02:50:13.046756 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:13.046792 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260927 02:50:13.162120 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:13.161899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:44631)
0927 02:50:13.162031 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:13.162067 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260927 02:50:13.518680 15687 raft_consensus.cc:482] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:13.518836 15687 raft_consensus.cc:504] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:13.519186 15687 leader_election.cc:288] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335), 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
Sep 27 02:50:13 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 02:50:13 dist-test-slave-f4rg krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 02:50:13.529186 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:13.519466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48467 (local address 127.0.98.3:33335)
0927 02:50:13.519624 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:13.519637 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:13.519671 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:50:13.520120 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:13.520122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:13.520163 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:13.520290 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:13.521108 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.521944 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.522960 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.523371 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.523465 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:13.525928 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:13.525933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:13.525933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:13.525946 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:13.527559 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:13.527963 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:13.527966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:13.527966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:13.528072 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:13.528351 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:13.528352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:13.528353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:13.528583 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:13.528648 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:13.528915 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 02:50:13.529077 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260927 02:50:13.529183 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:13.519515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37095 (local address 127.0.98.2:33281)
0927 02:50:13.519642 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:13.519651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:13.519680 (+ 29us) server_negotiation.cc:407] Connection header received
0927 02:50:13.520100 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:13.520102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:13.520163 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:13.520295 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:13.521111 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.521955 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.522948 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.523372 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.523470 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:13.526228 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:13.526233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:13.526234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:13.526245 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:13.527805 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:13.528170 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:13.528172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:13.528173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:13.528278 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:13.528446 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:13.528447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:13.528447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:13.528677 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:13.528745 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:13.528912 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 02:50:13.529068 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":46,"threads_started":1}
I20260927 02:50:13.529225 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:13.519360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33335 (local address 127.0.98.1:48467)
0927 02:50:13.519414 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:13.519427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:13.519487 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:13.520355 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:13.520358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:13.520776 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:13.520993 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:13.521000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.522039 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.522040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:13.522876 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:13.522879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.523454 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.523455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:13.523581 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:13.524167 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:13.524171 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:13.525793 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:13.527650 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:13.527657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:13.527659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:13.527888 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:13.528155 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:13.528156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:13.528156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:13.528278 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:13.528644 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:13.528650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:13.528771 (+ 121us) client_negotiation.cc:769] Sending connection context
0927 02:50:13.528856 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:50:13.529082 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":81}
I20260927 02:50:13.529330 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:13.519380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33281 (local address 127.0.98.1:37095)
0927 02:50:13.519583 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:13.519593 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:13.519638 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:13.520356 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:13.520359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:13.520775 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:13.520993 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:13.521000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.522049 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.522049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:13.522874 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:13.522876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:13.523457 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:13.523458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:13.523583 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:13.524128 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:13.524137 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:13.526131 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:13.527883 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:13.527884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:13.527885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:13.528064 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:13.528338 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:13.528339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:13.528339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:13.528395 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:13.528739 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:13.528740 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:13.528785 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:50:13.528856 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 02:50:13.529156 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":159,"thread_start_us":69,"threads_started":1}
W20260927 02:50:13.530076 15300 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
W20260927 02:50:13.530189 15302 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
I20260927 02:50:13.530241 15302 leader_election.cc:302] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389; no voters: 9d6c2e9a60f44aea9572361308f2cd2e, e0c9d8769a5345d1a71eac9d835c27e3
I20260927 02:50:13.530339 15687 raft_consensus.cc:2726] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:50:14.022598 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:14.022418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:39385)
0927 02:50:14.022513 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:14.022556 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 02:50:14.047538 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:14.047277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:44313)
0927 02:50:14.047406 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:14.047451 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260927 02:50:14.162880 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:14.162594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:41495)
0927 02:50:14.162751 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:14.162823 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260927 02:50:15.023392 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:15.023124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:52907)
0927 02:50:15.023295 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:15.023344 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260927 02:50:15.048167 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:15.047962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:38157)
0927 02:50:15.048081 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:15.048122 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260927 02:50:15.163591 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:15.163367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:48127)
0927 02:50:15.163497 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:15.163547 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260927 02:50:15.391525 15696 raft_consensus.cc:482] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:15.391611 15696 raft_consensus.cc:504] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:15.391747 15696 leader_election.cc:288] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335), 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
W20260927 02:50:15.392135 15300 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
W20260927 02:50:15.392215 15302 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
I20260927 02:50:15.392246 15302 leader_election.cc:302] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389; no voters: 9d6c2e9a60f44aea9572361308f2cd2e, e0c9d8769a5345d1a71eac9d835c27e3
I20260927 02:50:15.392313 15696 raft_consensus.cc:2726] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:50:16.024230 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:16.024000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:48213)
0927 02:50:16.024143 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:16.024183 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260927 02:50:16.048803 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:16.048580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:33995)
0927 02:50:16.048719 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:16.048761 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260927 02:50:16.164270 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:16.164072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:43101)
0927 02:50:16.164185 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:16.164228 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260927 02:50:17.024959 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:17.024743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:55825)
0927 02:50:17.024881 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:17.024916 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260927 02:50:17.049414 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:17.049226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:53097)
0927 02:50:17.049337 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:17.049374 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260927 02:50:17.155791 15702 raft_consensus.cc:482] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:17.155879 15702 raft_consensus.cc:504] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:17.156006 15702 leader_election.cc:288] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335), 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
W20260927 02:50:17.156360 15302 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
W20260927 02:50:17.156426 15300 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
I20260927 02:50:17.156513 15300 leader_election.cc:302] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389; no voters: 9d6c2e9a60f44aea9572361308f2cd2e, e0c9d8769a5345d1a71eac9d835c27e3
I20260927 02:50:17.156596 15702 raft_consensus.cc:2726] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:50:17.164830 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:17.164655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:47763)
0927 02:50:17.164756 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:17.164790 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260927 02:50:18.025686 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:18.025435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:59019)
0927 02:50:18.025591 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:18.025638 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260927 02:50:18.049984 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:18.049798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:51683)
0927 02:50:18.049915 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:18.049943 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260927 02:50:18.165409 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:18.165183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:47689)
0927 02:50:18.165325 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:18.165361 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260927 02:50:19.026376 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:19.026151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:48297)
0927 02:50:19.026289 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:19.026328 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260927 02:50:19.050611 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:19.050404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:50923)
0927 02:50:19.050529 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:19.050567 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260927 02:50:19.166030 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:19.165842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:39867)
0927 02:50:19.165950 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:19.165989 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260927 02:50:20.027158 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:20.026934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:55745)
0927 02:50:20.027066 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:20.027110 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260927 02:50:20.051178 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:20.050992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:44181)
0927 02:50:20.051099 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:20.051130 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260927 02:50:20.166594 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:20.166388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:56153)
0927 02:50:20.166514 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:20.166549 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260927 02:50:20.641055 15713 raft_consensus.cc:482] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:20.641147 15713 raft_consensus.cc:504] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:20.641300 15713 leader_election.cc:288] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335), 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
W20260927 02:50:20.641623 15300 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
W20260927 02:50:20.641683 15302 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
I20260927 02:50:20.641769 15302 leader_election.cc:302] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389; no voters: 9d6c2e9a60f44aea9572361308f2cd2e, e0c9d8769a5345d1a71eac9d835c27e3
I20260927 02:50:20.641850 15713 raft_consensus.cc:2726] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:50:21.027870 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:21.027635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:49725)
0927 02:50:21.027786 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:21.027824 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260927 02:50:21.051785 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:21.051558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:47887)
0927 02:50:21.051707 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:21.051742 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260927 02:50:21.167277 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:21.167078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:55133)
0927 02:50:21.167193 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:21.167233 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260927 02:50:22.028635 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:22.028383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:34693)
0927 02:50:22.028533 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:22.028577 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260927 02:50:22.052426 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:22.052198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:45185)
0927 02:50:22.052348 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:22.052383 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260927 02:50:22.167925 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:22.167725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:35849)
0927 02:50:22.167845 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:22.167881 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260927 02:50:23.029260 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:23.029040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:42487)
0927 02:50:23.029183 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:23.029214 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260927 02:50:23.053073 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:23.052859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:47261)
0927 02:50:23.052989 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:23.053026 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260927 02:50:23.168583 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:23.168382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:52487)
0927 02:50:23.168496 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:23.168536 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260927 02:50:24.029975 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:24.029696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:59955)
0927 02:50:24.029878 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:24.029923 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260927 02:50:24.053684 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:24.053490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:57167)
0927 02:50:24.053603 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:24.053641 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260927 02:50:24.083621 15725 raft_consensus.cc:482] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:24.083717 15725 raft_consensus.cc:504] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0c9d8769a5345d1a71eac9d835c27e3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33335 } } peers { permanent_uuid: "9d6c2e9a60f44aea9572361308f2cd2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33281 } } peers { permanent_uuid: "6a1b7b98551e431383cc1c9696b9e389" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36257 } }
I20260927 02:50:24.083851 15725 leader_election.cc:288] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335), 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281)
W20260927 02:50:24.084189 15302 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0c9d8769a5345d1a71eac9d835c27e3 (127.0.98.3:33335): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
W20260927 02:50:24.084261 15300 leader_election.cc:341] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d6c2e9a60f44aea9572361308f2cd2e (127.0.98.2:33281): Not found: Tablet not found: 99de37630f6e43db98d845311cbf6b09
I20260927 02:50:24.084311 15300 leader_election.cc:302] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a1b7b98551e431383cc1c9696b9e389; no voters: 9d6c2e9a60f44aea9572361308f2cd2e, e0c9d8769a5345d1a71eac9d835c27e3
I20260927 02:50:24.084378 15725 raft_consensus.cc:2726] T 99de37630f6e43db98d845311cbf6b09 P 6a1b7b98551e431383cc1c9696b9e389 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 02:50:24.169279 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:24.169025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:36287)
0927 02:50:24.169185 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:24.169225 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260927 02:50:25.030652 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:25.030445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:50031)
0927 02:50:25.030568 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:25.030608 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260927 02:50:25.054312 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:25.054077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:60301)
0927 02:50:25.054226 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:25.054266 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260927 02:50:25.169975 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:25.169744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:42681)
0927 02:50:25.169877 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:25.169918 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260927 02:50:26.031311 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:26.031104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:48591)
0927 02:50:26.031224 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:26.031265 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260927 02:50:26.054997 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:26.054767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:39767)
0927 02:50:26.054910 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:26.054951 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260927 02:50:26.170750 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:26.170507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:55199)
0927 02:50:26.170657 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:26.170702 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":65,"threads_started":1}
I20260927 02:50:27.032032 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:27.031826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:48335)
0927 02:50:27.031947 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:27.031988 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260927 02:50:27.055683 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:27.055457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:45261)
0927 02:50:27.055600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:27.055638 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260927 02:50:27.161294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 15255
I20260927 02:50:27.165618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 15417
I20260927 02:50:27.169467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 15551
2026-09-27T02:50:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18201 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:50:27 dist-test-slave-f4rg krb5kdc[15737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:50:27 dist-test-slave-f4rg krb5kdc[15737](info): set up 2 sockets
Sep 27 02:50:27 dist-test-slave-f4rg krb5kdc[15737](info): commencing operation
krb5kdc: starting...
W20260927 02:50:28.118673 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:50:28.232970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43285
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:28.312388 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:28.312602 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:28.312625 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:28.313737 15750 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:50:28.313769 15750 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:50:28.313781 15750 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:50:28.313791 15750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:28.313802 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:28.313813 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:28.313827 15750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:28.313838 15750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:28.315289 15750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:28.315491 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:28.315672 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:50:28.320475 15750 server_base.cc:1034] running on GCE node
W20260927 02:50:28.320510 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:28.320600 15755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:28.320520 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:28.320986 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:28.321244 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:28.322400 15750 hybrid_clock.cc:648] HybridClock initialized: now 1790477428322385 us; error 30 us; skew 500 ppm
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:28.324783 15750 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:28.325222 15750 webserver.cc:460] Webserver started at http://127.0.98.62:46405/ using document root <none> and password file <none>
I20260927 02:50:28.325428 15750 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:28.325474 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:28.325603 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:28.326666 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "a246b664d53b4b74a4d889e55940dc9d"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "fe397ae837d33c38892d355bbbeea355"
server_key_iv: "69aeaae1404221603e9b2e2894449078"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.327010 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "a246b664d53b4b74a4d889e55940dc9d"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "fe397ae837d33c38892d355bbbeea355"
server_key_iv: "69aeaae1404221603e9b2e2894449078"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.328315 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:50:28.329001 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.329155 15750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:28.329236 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "a246b664d53b4b74a4d889e55940dc9d"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "fe397ae837d33c38892d355bbbeea355"
server_key_iv: "69aeaae1404221603e9b2e2894449078"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.329300 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:28.339282 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:28.341024 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:28.341152 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:28.344864 15750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43285
I20260927 02:50:28.344906 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43285 every 8 connection(s)
I20260927 02:50:28.345223 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:28.346019 15818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:28.347358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15750
I20260927 02:50:28.347476 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:50:28.347694 392 external_mini_cluster.cc:1428] Setting key d41350c21df91612a3071f7191c4897f
I20260927 02:50:28.348730 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Bootstrap starting.
I20260927 02:50:28.349397 15818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:28.349659 15818 log.cc:824] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:28.350323 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: No bootstrap required, opened a new log
I20260927 02:50:28.352402 15818 raft_consensus.cc:348] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:28.352582 15818 raft_consensus.cc:374] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:28.352620 15818 raft_consensus.cc:729] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a246b664d53b4b74a4d889e55940dc9d, State: Initialized, Role: FOLLOWER
I20260927 02:50:28.352746 15818 consensus_queue.cc:260] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:28.352807 15818 raft_consensus.cc:388] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:28.352826 15818 raft_consensus.cc:482] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:28.352869 15818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:28.353557 15818 raft_consensus.cc:504] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:28.353669 15818 leader_election.cc:302] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a246b664d53b4b74a4d889e55940dc9d; no voters:
I20260927 02:50:28.353821 15818 leader_election.cc:288] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:50:28.353863 15823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:28.354068 15818 sys_catalog.cc:564] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:28.354105 15823 raft_consensus.cc:686] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 LEADER]: Becoming Leader. State: Replica: a246b664d53b4b74a4d889e55940dc9d, State: Running, Role: LEADER
I20260927 02:50:28.354282 15823 consensus_queue.cc:237] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:28.354614 15825 sys_catalog.cc:455] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a246b664d53b4b74a4d889e55940dc9d. Latest consensus state: current_term: 1 leader_uuid: "a246b664d53b4b74a4d889e55940dc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } } }
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:28.354602 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a246b664d53b4b74a4d889e55940dc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } } }
I20260927 02:50:28.354693 15825 sys_catalog.cc:458] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:28.354698 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:28.355050 15831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:28.355701 15831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:28.357769 15831 catalog_manager.cc:1347] Generated new cluster ID: e03d5dd9ca9b4a138744f70a1881a22b
I20260927 02:50:28.357817 15831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:50:28.359057 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.348215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.0.98.62:43285)
0927 02:50:28.348399 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.348410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.348440 (+ 30us) server_negotiation.cc:407] Connection header received
0927 02:50:28.348924 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.348932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.348991 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.349173 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:28.349749 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.351072 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.351776 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.352407 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.352523 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.355267 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:28.355271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:28.355272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:28.355284 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:28.357195 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.357518 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.357521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.357522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.357661 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.357860 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.357861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.357861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.358163 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:28.358289 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.358692 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.358846 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260927 02:50:28.367027 15831 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:50:28.367548 15831 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:28.373183 15831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Generated new TSK 0
I20260927 02:50:28.373365 15831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:50:28.403415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43285
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:28.519665 15844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:28.519934 15844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:28.519966 15844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:28.521610 15844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:28.521667 15844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:28.521695 15844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:50:28.523196 15844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:28.523401 15844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:28.523584 15844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:28.528505 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:28.528517 15852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:28.528615 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:28.528999 15844 server_base.cc:1034] running on GCE node
I20260927 02:50:28.529220 15844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:28.529448 15844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:28.530579 15844 hybrid_clock.cc:648] HybridClock initialized: now 1790477428530560 us; error 37 us; skew 500 ppm
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:28.533003 15844 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:50:28.533488 15844 webserver.cc:460] Webserver started at http://127.0.98.1:37661/ using document root <none> and password file <none>
I20260927 02:50:28.533697 15844 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:28.533746 15844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:28.533869 15844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:28.534961 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "aaf4e918d35a436dbf94391bc7d3191e"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "5f912379f9ff877673705fa34bc2a709"
server_key_iv: "6911cfe66840fec9330b05dc4a9bc6fa"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.535302 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aaf4e918d35a436dbf94391bc7d3191e"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "5f912379f9ff877673705fa34bc2a709"
server_key_iv: "6911cfe66840fec9330b05dc4a9bc6fa"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.536582 15844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:50:28.537288 15859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.537439 15844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:28.537526 15844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "aaf4e918d35a436dbf94391bc7d3191e"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "5f912379f9ff877673705fa34bc2a709"
server_key_iv: "6911cfe66840fec9330b05dc4a9bc6fa"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.537592 15844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:28.558976 15844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:28.560850 15844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:28.560986 15844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:28.561199 15844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:28.561513 15844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:28.561547 15844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.561576 15844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:28.561594 15844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.566987 15844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45709
I20260927 02:50:28.567066 15972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45709 every 8 connection(s)
I20260927 02:50:28.567343 15844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:50:28.568993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15844
I20260927 02:50:28.569087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:50:28.569257 392 external_mini_cluster.cc:1428] Setting key 75bb0953d3d5ad5c595a758961e88d23
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:50:28.577317 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.568005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42793 (local address 127.0.98.62:43285)
0927 02:50:28.568056 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.568057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.568254 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:50:28.568755 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.568756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.568768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.568815 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:28.569873 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.570586 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.571889 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.572093 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.572155 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.574555 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:28.574556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:28.574557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:28.574563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:28.575929 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.576377 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.576378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.576378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.576424 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.576735 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.576735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.576735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.576834 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:28.576879 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.577182 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.577251 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:50:28.577553 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.567993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:42793)
0927 02:50:28.568141 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.568175 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.568664 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.568912 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.568915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.569585 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:28.569791 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.569795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.570669 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.570670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.571781 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.571784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.572178 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.572179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.572315 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.572819 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:28.572828 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:28.574438 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:28.576011 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:28.576019 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:28.576021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:28.576306 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:28.576525 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:28.576526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:28.576526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:28.576676 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:28.576885 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:28.576890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:28.577056 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.577169 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.577334 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260927 02:50:28.578017 15973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43285
I20260927 02:50:28.578109 15973 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:28.578294 15973 heartbeater.cc:511] Master 127.0.98.62:43285 requested a full tablet report, sending...
I20260927 02:50:28.578830 15782 ts_manager.cc:194] Registered new tserver with Master: aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709)
I20260927 02:50:28.579442 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:50:28.593466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43285
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:28.673890 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:28.674098 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:28.674121 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:28.675277 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:28.675319 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:28.675348 15978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:50:28.676759 15978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:28.676967 15978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:28.677179 15978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:28.681794 15983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:28.681794 15986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:28.681797 15984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:28.682080 15978 server_base.cc:1034] running on GCE node
I20260927 02:50:28.682303 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:28.682534 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:28.683666 15978 hybrid_clock.cc:648] HybridClock initialized: now 1790477428683649 us; error 30 us; skew 500 ppm
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:28.686030 15978 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:50:28.686458 15978 webserver.cc:460] Webserver started at http://127.0.98.2:45507/ using document root <none> and password file <none>
I20260927 02:50:28.686655 15978 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:28.686702 15978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:28.686852 15978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:28.687883 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "862d02e84725424d9a6fdddfc26e41a3"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "409025fbe4c270a5e40357bd34526f8f"
server_key_iv: "6b56353a064bb303344912535f84bef1"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.688215 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "862d02e84725424d9a6fdddfc26e41a3"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "409025fbe4c270a5e40357bd34526f8f"
server_key_iv: "6b56353a064bb303344912535f84bef1"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.689395 15978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:50:28.690025 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.690171 15978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:50:28.690250 15978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "862d02e84725424d9a6fdddfc26e41a3"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "409025fbe4c270a5e40357bd34526f8f"
server_key_iv: "6b56353a064bb303344912535f84bef1"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.690311 15978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:28.707695 15978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:28.709414 15978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:28.709546 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:28.709762 15978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:28.710043 15978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:28.710074 15978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.710110 15978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:28.710135 15978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.715664 15978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34325
I20260927 02:50:28.715720 16106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34325 every 8 connection(s)
I20260927 02:50:28.716033 15978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:50:28.717869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 15978
I20260927 02:50:28.717979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:50:28.718190 392 external_mini_cluster.cc:1428] Setting key 6aba0fd1cee85a8fce297d971e7845a5
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:50:28.725850 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.716804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42211 (local address 127.0.98.62:43285)
0927 02:50:28.716868 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.716870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.717078 (+ 208us) server_negotiation.cc:407] Connection header received
0927 02:50:28.717544 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.717545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.717555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.717650 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:28.718846 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.719380 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.720665 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.720943 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.721012 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.723154 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:28.723156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:28.723156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:28.723165 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:28.724573 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.724948 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.724949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.724949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.725001 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.725217 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.725218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.725218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.725324 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:28.725379 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.725662 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.725761 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:50:28.726065 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.716762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:42211)
0927 02:50:28.716933 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.716973 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.717458 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.717742 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.717747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.718501 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:28.718728 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.718736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.719469 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.719470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.720559 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.720562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.721031 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.721032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.721134 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.721639 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:28.721647 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:28.723066 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:28.724654 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:28.724660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:28.724662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:28.724868 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:28.725059 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:28.725060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:28.725060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:28.725162 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:28.725378 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:28.725384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:28.725526 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.725626 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.725783 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260927 02:50:28.726508 16107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43285
I20260927 02:50:28.726598 16107 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:28.726763 16107 heartbeater.cc:511] Master 127.0.98.62:43285 requested a full tablet report, sending...
I20260927 02:50:28.727128 15782 ts_manager.cc:194] Registered new tserver with Master: 862d02e84725424d9a6fdddfc26e41a3 (127.0.98.2:34325)
I20260927 02:50:28.727501 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:50:28.741372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43285
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:28.819576 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:28.819792 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:28.819813 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:28.820972 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:28.821017 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:28.821043 16112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:50:28.822492 16112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:28.822721 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:28.822957 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:28.827565 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:28.827625 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:28.827781 16112 server_base.cc:1034] running on GCE node
W20260927 02:50:28.827561 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:28.828048 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:28.828272 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:28.829406 16112 hybrid_clock.cc:648] HybridClock initialized: now 1790477428829394 us; error 25 us; skew 500 ppm
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:28.831791 16112 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:50:28.832252 16112 webserver.cc:460] Webserver started at http://127.0.98.3:43799/ using document root <none> and password file <none>
I20260927 02:50:28.832461 16112 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:28.832520 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:28.832628 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:50:28.833638 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c19b38dea454848b6e314eb1c7116d8"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "df95e816afe2ddbbdadf18548018e11c"
server_key_iv: "78e6333777422da50adebce26ca264c4"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.833918 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c19b38dea454848b6e314eb1c7116d8"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "df95e816afe2ddbbdadf18548018e11c"
server_key_iv: "78e6333777422da50adebce26ca264c4"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.835088 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:50:28.835776 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.835916 16112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.835988 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "5c19b38dea454848b6e314eb1c7116d8"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "df95e816afe2ddbbdadf18548018e11c"
server_key_iv: "78e6333777422da50adebce26ca264c4"
server_key_version: "encryptionkey@0"
I20260927 02:50:28.836052 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:28.851737 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:28.853498 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:28.853626 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:28.853824 16112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:28.854128 16112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:28.854166 16112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.854203 16112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:28.854228 16112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:28.859866 16112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34325
I20260927 02:50:28.859946 16240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34325 every 8 connection(s)
I20260927 02:50:28.860221 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:50:28.865692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 16112
I20260927 02:50:28.865779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
Sep 27 02:50:28 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:28.865955 392 external_mini_cluster.cc:1428] Setting key f5bfc23c85c8f791f0f5327eaa32cb36
I20260927 02:50:28.868819 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.860831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53659 (local address 127.0.98.62:43285)
0927 02:50:28.860890 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.860892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.861093 (+ 201us) server_negotiation.cc:407] Connection header received
0927 02:50:28.861620 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.861621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.861632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.861683 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:28.862594 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.863145 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.864097 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.864298 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.864351 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.866379 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:28.866381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:28.866381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:28.866397 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:28.867627 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.867969 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.867970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.867971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.868026 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.868236 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.868237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.868237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.868349 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:28.868405 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.868652 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.868729 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:50:28.868988 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.860820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:53659)
0927 02:50:28.860976 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.861009 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.861483 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.861752 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.861756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.862306 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:28.862499 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.862503 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.863228 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.863229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.864013 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.864015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.864361 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.864362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.864474 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.864898 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:28.864906 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:28.866260 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:28.867696 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:28.867702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:28.867705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:28.867907 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:28.868079 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:28.868080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:28.868080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:28.868184 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:28.868401 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:28.868406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:28.868540 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.868626 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.868778 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260927 02:50:28.869386 16241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43285
I20260927 02:50:28.869468 16241 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:28.869604 16241 heartbeater.cc:511] Master 127.0.98.62:43285 requested a full tablet report, sending...
I20260927 02:50:28.869910 15782 ts_manager.cc:194] Registered new tserver with Master: 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325)
I20260927 02:50:28.870280 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53659
I20260927 02:50:28.877399 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:50:28.884011 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.878653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.0.98.62:43285)
0927 02:50:28.878704 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.878706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.878737 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:50:28.878839 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.878840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.878849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.878891 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:28.879390 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.879905 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.880603 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.880799 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.880851 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.881896 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:28.881897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:28.881898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:28.881905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:28.882990 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.883282 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.883283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.883283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.883336 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:28.883488 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:28.883489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:28.883489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:28.883642 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:28.883727 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.883844 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.883922 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:50:28.885373 15782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:50:28.886930 15782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:50:28.892948 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.888991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45709 (local address 127.0.0.1:52094)
0927 02:50:28.889338 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.889345 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.889381 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.889564 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.889568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.889570 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:28.889637 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.889639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.890616 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.890617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.891901 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.891904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.892637 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.892638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.892712 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.892725 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.892788 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.892839 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":90,"threads_started":1}
I20260927 02:50:28.892963 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.889233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.0.98.1:45709)
0927 02:50:28.889373 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.889378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.889390 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:50:28.889435 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.889438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.889454 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.889550 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:28.889714 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.890525 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.891992 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.892567 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.892629 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.892690 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.892808 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.892885 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260927 02:50:28.893208 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.889173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.0.98.2:34325)
0927 02:50:28.889422 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.889426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.889434 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:28.889466 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.889468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.889483 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.889550 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:28.889739 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.890883 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.892127 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.892782 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.892835 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.892883 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.893085 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.893135 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":72,"threads_started":1}
I20260927 02:50:28.893265 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.889170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.0.98.3:34325)
0927 02:50:28.889485 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.889489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.889497 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:28.889529 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.889531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.889545 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.889651 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:28.889897 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.890967 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.891957 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.892670 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.892736 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.892790 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.893152 (+ 362us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.893191 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":69,"threads_started":1}
I20260927 02:50:28.893049 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.888983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34325 (local address 127.0.0.1:46180)
0927 02:50:28.889144 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.889184 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.889247 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.889782 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.889783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.889785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:28.889825 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.889826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.891039 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.891040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.891888 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.891891 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.892823 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.892825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.892891 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.892895 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.892937 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.892974 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":35,"thread_start_us":47,"threads_started":1}
I20260927 02:50:28.893368 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.888983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34325 (local address 127.0.0.1:35346)
0927 02:50:28.889310 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.889319 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.889363 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.889631 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.889632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.889634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:28.889678 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.889679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.890992 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.890993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.892055 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.892056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.892910 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.892911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.892964 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.892967 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.893007 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.893036 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":7,"thread_start_us":78,"threads_started":1}
I20260927 02:50:28.894115 16041 tablet_service.cc:1467] Processing CreateTablet for tablet 9981b4e4e01f4f4e9f1069130e5acfef (DEFAULT_TABLE table=test-table [id=f8b782d59e264ecea3a4c4069e586f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:28.894114 16175 tablet_service.cc:1467] Processing CreateTablet for tablet 9981b4e4e01f4f4e9f1069130e5acfef (DEFAULT_TABLE table=test-table [id=f8b782d59e264ecea3a4c4069e586f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:28.894124 15907 tablet_service.cc:1467] Processing CreateTablet for tablet 9981b4e4e01f4f4e9f1069130e5acfef (DEFAULT_TABLE table=test-table [id=f8b782d59e264ecea3a4c4069e586f2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:28.894456 15907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981b4e4e01f4f4e9f1069130e5acfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:28.894457 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981b4e4e01f4f4e9f1069130e5acfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:28.894457 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9981b4e4e01f4f4e9f1069130e5acfef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:28.897083 16258 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap starting.
I20260927 02:50:28.897509 16259 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Bootstrap starting.
I20260927 02:50:28.897749 16260 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Bootstrap starting.
I20260927 02:50:28.897789 16258 tablet_bootstrap.cc:654] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:28.898183 16259 tablet_bootstrap.cc:654] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:28.898293 16260 tablet_bootstrap.cc:654] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:28.898304 16258 log.cc:824] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:28.898644 16259 log.cc:824] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:28.898823 16260 log.cc:824] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:28.899062 16258 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: No bootstrap required, opened a new log
I20260927 02:50:28.899132 16258 ts_tablet_manager.cc:1397] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:28.899358 16259 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: No bootstrap required, opened a new log
I20260927 02:50:28.899420 16259 ts_tablet_manager.cc:1397] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:50:28.899578 16260 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: No bootstrap required, opened a new log
I20260927 02:50:28.899636 16260 ts_tablet_manager.cc:1397] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:28.901008 16258 raft_consensus.cc:348] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.901082 16260 raft_consensus.cc:348] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.901188 16258 raft_consensus.cc:374] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:28.901208 16260 raft_consensus.cc:374] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:28.901224 16258 raft_consensus.cc:729] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf4e918d35a436dbf94391bc7d3191e, State: Initialized, Role: FOLLOWER
I20260927 02:50:28.901268 16260 raft_consensus.cc:729] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862d02e84725424d9a6fdddfc26e41a3, State: Initialized, Role: FOLLOWER
I20260927 02:50:28.901353 16258 consensus_queue.cc:260] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.901391 16260 consensus_queue.cc:260] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.901451 16259 raft_consensus.cc:348] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.901605 16259 raft_consensus.cc:374] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:28.901628 16260 ts_tablet_manager.cc:1428] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:50:28.901631 16258 ts_tablet_manager.cc:1428] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 02:50:28.901645 16259 raft_consensus.cc:729] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c19b38dea454848b6e314eb1c7116d8, State: Initialized, Role: FOLLOWER
I20260927 02:50:28.901755 15973 heartbeater.cc:503] Master 127.0.98.62:43285 was elected leader, sending a full tablet report...
I20260927 02:50:28.901739 16259 consensus_queue.cc:260] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.901997 16241 heartbeater.cc:503] Master 127.0.98.62:43285 was elected leader, sending a full tablet report...
I20260927 02:50:28.902071 16107 heartbeater.cc:503] Master 127.0.98.62:43285 was elected leader, sending a full tablet report...
I20260927 02:50:28.901984 16259 ts_tablet_manager.cc:1428] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:50:28.939378 16265 raft_consensus.cc:482] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:28.939494 16265 raft_consensus.cc:504] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.939883 16265 leader_election.cc:288] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325), aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709)
I20260927 02:50:28.943152 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.940038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45709 (local address 127.0.98.2:46705)
0927 02:50:28.940087 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.940110 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.940153 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.940331 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.940332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.940333 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:28.940407 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.940409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.941053 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.941053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.942050 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.942052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.942856 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.942858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.942974 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.942978 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.943043 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.943085 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:50:28.943149 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.940131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36629 (local address 127.0.98.3:34325)
0927 02:50:28.940180 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.940182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.940313 (+ 131us) server_negotiation.cc:407] Connection header received
0927 02:50:28.940360 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.940361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.940374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.940427 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:28.940621 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.941136 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.942201 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.942772 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.942856 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.942865 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.943027 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.943072 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:50:28.943198 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.940162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46705 (local address 127.0.98.1:45709)
0927 02:50:28.940203 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.940205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.940213 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:28.940240 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.940241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.940253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.940309 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:28.940468 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.940982 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.942151 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.942728 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.942792 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.942824 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.943053 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.943119 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:50:28.943346 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.940038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34325 (local address 127.0.98.2:36629)
0927 02:50:28.940247 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.940258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:28.940309 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:28.940432 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:28.940433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:28.940434 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:28.940507 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.940508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.941204 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.941204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.942117 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:28.942120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.942875 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.942876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:28.942948 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.942952 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:28.943235 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 02:50:28.943276 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":38,"thread_start_us":114,"threads_started":1}
I20260927 02:50:28.943543 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b4e4e01f4f4e9f1069130e5acfef" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf4e918d35a436dbf94391bc7d3191e" is_pre_election: true
I20260927 02:50:28.943701 15927 raft_consensus.cc:2454] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 0.
I20260927 02:50:28.943727 16195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b4e4e01f4f4e9f1069130e5acfef" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c19b38dea454848b6e314eb1c7116d8" is_pre_election: true
I20260927 02:50:28.943863 16195 raft_consensus.cc:2454] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 0.
I20260927 02:50:28.943902 15997 leader_election.cc:302] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862d02e84725424d9a6fdddfc26e41a3, aaf4e918d35a436dbf94391bc7d3191e; no voters:
I20260927 02:50:28.944028 16265 raft_consensus.cc:2781] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:50:28.944135 16265 raft_consensus.cc:482] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:50:28.944177 16265 raft_consensus.cc:3037] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:28.945051 16265 raft_consensus.cc:504] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.945165 16265 leader_election.cc:288] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 election: Requested vote from peers 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325), aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709)
I20260927 02:50:28.945303 16195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b4e4e01f4f4e9f1069130e5acfef" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c19b38dea454848b6e314eb1c7116d8"
I20260927 02:50:28.945314 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9981b4e4e01f4f4e9f1069130e5acfef" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf4e918d35a436dbf94391bc7d3191e"
I20260927 02:50:28.945377 16195 raft_consensus.cc:3037] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:28.945377 15927 raft_consensus.cc:3037] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:28.946203 16195 raft_consensus.cc:2454] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 1.
I20260927 02:50:28.946203 15927 raft_consensus.cc:2454] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 1.
I20260927 02:50:28.946352 15997 leader_election.cc:302] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 862d02e84725424d9a6fdddfc26e41a3, aaf4e918d35a436dbf94391bc7d3191e; no voters:
I20260927 02:50:28.946451 16265 raft_consensus.cc:2781] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:28.946601 16265 raft_consensus.cc:686] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 1 LEADER]: Becoming Leader. State: Replica: 862d02e84725424d9a6fdddfc26e41a3, State: Running, Role: LEADER
I20260927 02:50:28.946686 16265 consensus_queue.cc:237] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:28.947515 15781 catalog_manager.cc:5495] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 862d02e84725424d9a6fdddfc26e41a3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "862d02e84725424d9a6fdddfc26e41a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } health_report { overall_health: UNKNOWN } } }
I20260927 02:50:28.964825 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:28.962749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.0.98.2:34325)
0927 02:50:28.962798 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:28.962799 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:28.962826 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:50:28.962867 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:28.962868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:28.962891 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:28.962950 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:50:28.963070 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.963545 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.964252 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:28.964426 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:28.964468 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:28.964572 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:50:28.964665 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:50:28.964719 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:28.964735 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 02:50:28.964750 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 02:50:28.966522 16108 tablet.cc:2369] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:50:28.966555 16195 raft_consensus.cc:1261] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Refusing update from remote peer 862d02e84725424d9a6fdddfc26e41a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:28.966568 15927 raft_consensus.cc:1261] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Refusing update from remote peer 862d02e84725424d9a6fdddfc26e41a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:28.966919 16268 consensus_queue.cc:1035] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:28.967047 16265 consensus_queue.cc:1035] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:28.969317 16271 mvcc.cc:205] Tried to move back new op lower bound from 7333795549044285440 to 7333795548966313984. Current Snapshot: MvccSnapshot[applied={T|T < 7333795549044285440}]
I20260927 02:50:28.969432 16272 mvcc.cc:205] Tried to move back new op lower bound from 7333795549044285440 to 7333795548966313984. Current Snapshot: MvccSnapshot[applied={T|T < 7333795549044285440}]
I20260927 02:50:28.969947 16273 mvcc.cc:205] Tried to move back new op lower bound from 7333795549044285440 to 7333795548966313984. Current Snapshot: MvccSnapshot[applied={T|T < 7333795549044285440 or (T in {7333795549044285440})}]
I20260927 02:50:28.971525 15782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:28.971612 15782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:28.972270 15782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Sending DeleteTablet for 3 replicas of tablet 9981b4e4e01f4f4e9f1069130e5acfef
I20260927 02:50:28.972550 16041 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981b4e4e01f4f4e9f1069130e5acfef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35346
I20260927 02:50:28.972604 16175 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981b4e4e01f4f4e9f1069130e5acfef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46180
I20260927 02:50:28.972615 15907 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981b4e4e01f4f4e9f1069130e5acfef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52094
I20260927 02:50:28.972759 16281 tablet_replica.cc:330] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: stopping tablet replica
I20260927 02:50:28.972822 16283 tablet_replica.cc:330] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: stopping tablet replica
I20260927 02:50:28.972841 16281 raft_consensus.cc:2229] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:50:28.972851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 15844
I20260927 02:50:28.972987 16283 raft_consensus.cc:2229] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:28.973125 16283 raft_consensus.cc:2258] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:28.973408 16281 raft_consensus.cc:2258] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:28.973555 16282 tablet_replica.cc:330] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: stopping tablet replica
I20260927 02:50:28.973798 16282 raft_consensus.cc:2229] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:28.973866 16281 ts_tablet_manager.cc:1905] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:28.973934 16282 raft_consensus.cc:2258] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:28.974357 16282 ts_tablet_manager.cc:1905] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:28.975411 16281 ts_tablet_manager.cc:1918] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:28.975541 16281 log.cc:1196] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/9981b4e4e01f4f4e9f1069130e5acfef
I20260927 02:50:28.975853 16281 ts_tablet_manager.cc:1939] T 9981b4e4e01f4f4e9f1069130e5acfef P 862d02e84725424d9a6fdddfc26e41a3: Deleting consensus metadata
I20260927 02:50:28.975885 16282 ts_tablet_manager.cc:1918] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:28.976011 16282 log.cc:1196] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/9981b4e4e01f4f4e9f1069130e5acfef
I20260927 02:50:28.976307 16282 ts_tablet_manager.cc:1939] T 9981b4e4e01f4f4e9f1069130e5acfef P 5c19b38dea454848b6e314eb1c7116d8: Deleting consensus metadata
I20260927 02:50:28.976297 15768 catalog_manager.cc:4841] TS 862d02e84725424d9a6fdddfc26e41a3 (127.0.98.2:34325): tablet 9981b4e4e01f4f4e9f1069130e5acfef (table test-table [id=f8b782d59e264ecea3a4c4069e586f2c]) successfully deleted
I20260927 02:50:28.976756 15767 catalog_manager.cc:4841] TS 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325): tablet 9981b4e4e01f4f4e9f1069130e5acfef (table test-table [id=f8b782d59e264ecea3a4c4069e586f2c]) successfully deleted
W20260927 02:50:28.977929 15769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45709 (error 108)
W20260927 02:50:28.979041 16254 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:50:28.978915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45709 (local address 127.0.0.1:52102)
0927 02:50:28.978964 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:28.979002 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45709: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:50:28.979215 15769 catalog_manager.cc:4584] TS aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9981b4e4e01f4f4e9f1069130e5acfef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45709: connect: Connection reset by peer (error 104)
I20260927 02:50:28.979384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 15978
I20260927 02:50:28.983748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 16112
I20260927 02:50:28.988111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 15750
I20260927 02:50:48.992725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43285
--webserver_interface=127.0.98.62
--webserver_port=46405
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:50:49.074265 16285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:49.074481 16285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:49.074503 16285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:49.075700 16285 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:50:49.075745 16285 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:50:49.075757 16285 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:50:49.075768 16285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:50:49.075780 16285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:49.075791 16285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:49.075804 16285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:50:49.075827 16285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:50:49.077382 16285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:49.077632 16285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:49.077929 16285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:49.086159 16294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:49.086156 16291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:49.086187 16292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:49.086551 16285 server_base.cc:1034] running on GCE node
I20260927 02:50:49.086853 16285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:49.087194 16285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:49.088341 16285 hybrid_clock.cc:648] HybridClock initialized: now 1790477449088324 us; error 36 us; skew 500 ppm
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:49.095192 16285 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:50:49.095679 16285 webserver.cc:460] Webserver started at http://127.0.98.62:46405/ using document root <none> and password file <none>
I20260927 02:50:49.095908 16285 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:49.095983 16285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:49.097338 16285 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 02:50:49.098001 16301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.098148 16285 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.098233 16285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "a246b664d53b4b74a4d889e55940dc9d"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "fe397ae837d33c38892d355bbbeea355"
server_key_iv: "69aeaae1404221603e9b2e2894449078"
server_key_version: "encryptionkey@0"
I20260927 02:50:49.098628 16285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:49.121050 16285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:49.123068 16285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:49.123211 16285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:49.126959 16285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43285
I20260927 02:50:49.127002 16353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43285 every 8 connection(s)
I20260927 02:50:49.127365 16285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:50:49.127750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 16285
I20260927 02:50:49.128010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45709
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43285
--webserver_port=37661
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:49.128947 16354 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:50:49.129593 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Bootstrap starting.
I20260927 02:50:49.137960 16354 log.cc:824] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:49.139956 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:50:49.140239 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Bootstrap complete.
I20260927 02:50:49.142385 16354 raft_consensus.cc:348] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:49.142642 16354 raft_consensus.cc:729] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a246b664d53b4b74a4d889e55940dc9d, State: Initialized, Role: FOLLOWER
I20260927 02:50:49.142820 16354 consensus_queue.cc:260] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:49.142897 16354 raft_consensus.cc:388] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:50:49.142951 16354 raft_consensus.cc:482] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:50:49.143009 16354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:50:49.144039 16354 raft_consensus.cc:504] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:49.144134 16354 leader_election.cc:302] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a246b664d53b4b74a4d889e55940dc9d; no voters:
I20260927 02:50:49.144300 16354 leader_election.cc:288] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 02:50:49.144349 16358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:50:49.144493 16358 raft_consensus.cc:686] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 LEADER]: Becoming Leader. State: Replica: a246b664d53b4b74a4d889e55940dc9d, State: Running, Role: LEADER
I20260927 02:50:49.144579 16354 sys_catalog.cc:564] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:50:49.144618 16358 consensus_queue.cc:237] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:50:49.144881 16358 sys_catalog.cc:455] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a246b664d53b4b74a4d889e55940dc9d. Latest consensus state: current_term: 2 leader_uuid: "a246b664d53b4b74a4d889e55940dc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } } }
I20260927 02:50:49.145115 16358 sys_catalog.cc:458] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:49.145048 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a246b664d53b4b74a4d889e55940dc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } } }
I20260927 02:50:49.145524 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:50:49.145678 16367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:50:49.146355 16367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:50:49.146432 16367 catalog_manager.cc:1259] Loaded cluster ID: e03d5dd9ca9b4a138744f70a1881a22b
I20260927 02:50:49.146461 16367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:50:49.147290 16367 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:50:49.147477 16367 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Loaded TSK: 0
I20260927 02:50:49.161895 16367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Generated new TSK 1
I20260927 02:50:49.162009 16367 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:50:49.222958 16356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:49.223173 16356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:49.223206 16356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:49.224386 16356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:49.224429 16356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:49.224464 16356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:50:49.226254 16356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37661
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:49.226558 16356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:49.226778 16356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:49.231539 16382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:49.231529 16381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:49.231742 16384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:49.231871 16356 server_base.cc:1034] running on GCE node
I20260927 02:50:49.232101 16356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:49.232407 16356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:49.233596 16356 hybrid_clock.cc:648] HybridClock initialized: now 1790477449233603 us; error 56 us; skew 500 ppm
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:49.236724 16356 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:50:49.237144 16356 webserver.cc:460] Webserver started at http://127.0.98.1:37661/ using document root <none> and password file <none>
I20260927 02:50:49.237349 16356 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:49.237409 16356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:49.238827 16356 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:49.239528 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.239689 16356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:50:49.239774 16356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "aaf4e918d35a436dbf94391bc7d3191e"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "5f912379f9ff877673705fa34bc2a709"
server_key_iv: "6911cfe66840fec9330b05dc4a9bc6fa"
server_key_version: "encryptionkey@0"
I20260927 02:50:49.240134 16356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:49.271384 16356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:49.273636 16356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:49.273783 16356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:49.274014 16356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:49.274438 16398 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:50:49.276474 16356 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:50:49.276537 16356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:50:49.276573 16356 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:50:49.277276 16356 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:50:49.277309 16356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:50:49.277398 16398 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap starting.
I20260927 02:50:49.284283 16356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45709
I20260927 02:50:49.284652 16356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:50:49.287508 16505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45709 every 8 connection(s)
I20260927 02:50:49.287953 16398 log.cc:824] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:49.290256 16398 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:50:49.290506 16398 tablet_bootstrap.cc:492] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap complete.
I20260927 02:50:49.291270 16398 ts_tablet_manager.cc:1397] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:49.292157 16398 raft_consensus.cc:348] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.292394 16398 raft_consensus.cc:729] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf4e918d35a436dbf94391bc7d3191e, State: Initialized, Role: FOLLOWER
I20260927 02:50:49.292526 16398 consensus_queue.cc:260] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.292780 16398 ts_tablet_manager.cc:1428] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 02:50:49.293102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 16356
I20260927 02:50:49.293388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34325
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43285
--webserver_port=45507
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:49.296434 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.285393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46699 (local address 127.0.98.62:43285)
0927 02:50:49.286972 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.286980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.287012 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:50:49.287121 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.287126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.287169 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.287289 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:49.288135 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.289152 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.290127 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.290551 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.290658 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.292808 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:49.292813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:49.292813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:49.292824 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:49.294615 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.295163 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.295165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.295166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.295258 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.295549 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.295550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.295550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.295797 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:49.295869 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.296187 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.296295 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1519,"thread_start_us":84,"threads_started":1}
I20260927 02:50:49.296505 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.285289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:46699)
0927 02:50:49.285465 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.285510 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.286042 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.287333 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.287337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.287733 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:49.287997 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.288003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.289260 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.289261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.290050 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.290052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.290649 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.290650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.290782 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.291313 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:49.291322 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:49.292721 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:49.294704 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.294712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.294715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.295085 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.295318 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.295319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.295320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.295448 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.295870 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:49.295875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:49.296039 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.296147 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.296305 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260927 02:50:49.297251 16506 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43285
I20260927 02:50:49.297351 16506 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:49.297537 16506 heartbeater.cc:511] Master 127.0.98.62:43285 requested a full tablet report, sending...
I20260927 02:50:49.298271 16318 ts_manager.cc:194] Registered new tserver with Master: aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709)
I20260927 02:50:49.299268 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46699
I20260927 02:50:49.300271 16506 heartbeater.cc:503] Master 127.0.98.62:43285 was elected leader, sending a full tablet report...
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:50:49.306782 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.298917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59572 (local address 127.0.98.1:45709)
0927 02:50:49.299538 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.299542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.299702 (+ 160us) server_negotiation.cc:407] Connection header received
0927 02:50:49.299818 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.299820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.299835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.299895 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:49.300313 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.301101 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.301939 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.302235 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.302334 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.304255 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:49.304263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:49.304263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:49.304270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:49.305662 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.306013 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.306016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.306016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.306070 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.306230 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.306231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.306232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.306362 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:49.306428 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.306598 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.306680 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":83,"threads_started":1}
I20260927 02:50:49.306782 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.298902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45709 (local address 127.0.0.1:59572)
0927 02:50:49.299606 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.299643 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.299735 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.299902 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.299906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.300197 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:49.300248 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.300250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.301168 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.301170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.301862 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.301865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.302332 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.302336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.302414 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.302947 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:49.302951 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:49.304176 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:49.305746 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.305752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.305754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.305938 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.306124 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.306125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.306125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.306183 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.306418 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:49.306425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:49.306485 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.306557 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.306689 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":86,"threads_started":1}
I20260927 02:50:49.307287 16440 tablet_service.cc:1514] Processing DeleteTablet for tablet 9981b4e4e01f4f4e9f1069130e5acfef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59572
I20260927 02:50:49.307789 16515 tablet_replica.cc:330] stopping tablet replica
I20260927 02:50:49.307880 16515 raft_consensus.cc:2229] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:49.307947 16515 raft_consensus.cc:2258] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:49.308236 16515 ts_tablet_manager.cc:1905] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:49.310245 16515 ts_tablet_manager.cc:1918] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:49.310303 16515 log.cc:1196] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal/wals/9981b4e4e01f4f4e9f1069130e5acfef
I20260927 02:50:49.310504 16515 ts_tablet_manager.cc:1939] T 9981b4e4e01f4f4e9f1069130e5acfef P aaf4e918d35a436dbf94391bc7d3191e: Deleting consensus metadata
I20260927 02:50:49.311002 16305 catalog_manager.cc:4841] TS aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709): tablet 9981b4e4e01f4f4e9f1069130e5acfef (table test-table [id=f8b782d59e264ecea3a4c4069e586f2c]) successfully deleted
W20260927 02:50:49.373807 16512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:49.374022 16512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:49.374053 16512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:49.375248 16512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:49.375293 16512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:49.375329 16512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:50:49.376775 16512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45507
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:49.377005 16512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:49.377213 16512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:49.382198 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:49.382237 16512 server_base.cc:1034] running on GCE node
W20260927 02:50:49.382205 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:49.382222 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:49.382555 16512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:49.382784 16512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:49.383975 16512 hybrid_clock.cc:648] HybridClock initialized: now 1790477449383959 us; error 35 us; skew 500 ppm
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:49.386301 16512 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:50:49.386691 16512 webserver.cc:460] Webserver started at http://127.0.98.2:45507/ using document root <none> and password file <none>
I20260927 02:50:49.386914 16512 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:49.386976 16512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:49.388260 16512 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:50:49.388747 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.388898 16512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.388974 16512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "862d02e84725424d9a6fdddfc26e41a3"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "409025fbe4c270a5e40357bd34526f8f"
server_key_iv: "6b56353a064bb303344912535f84bef1"
server_key_version: "encryptionkey@0"
I20260927 02:50:49.389283 16512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:49.401850 16512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:49.403983 16512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:49.404101 16512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:49.404289 16512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:49.404542 16512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:49.404579 16512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.404601 16512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:49.404613 16512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.409929 16512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34325
I20260927 02:50:49.409996 16643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34325 every 8 connection(s)
I20260927 02:50:49.410300 16512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:49.418708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 16512
I20260927 02:50:49.419047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34325
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43285
--webserver_port=43799
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:50:49.419433 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.410992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34265 (local address 127.0.98.62:43285)
0927 02:50:49.411079 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.411081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.411234 (+ 153us) server_negotiation.cc:407] Connection header received
0927 02:50:49.411778 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.411779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.411793 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.411849 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:49.412764 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.413258 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.414256 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.414466 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.414529 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.416696 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:49.416697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:49.416698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:49.416705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:49.417996 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.418370 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.418371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.418371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.418426 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.418697 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.418697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.418698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.418801 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:49.418894 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.419249 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.419340 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:50:49.419920 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.410975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:34265)
0927 02:50:49.411117 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.411153 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.411660 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.411908 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.411911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.412471 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:49.412681 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.412686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.413325 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.413325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.414153 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.414156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.414571 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.414572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.414688 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.415203 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:49.415211 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:49.416595 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:49.418078 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.418085 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.418087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.418300 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.418496 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.418496 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.418497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.418637 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.418885 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:49.418892 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:49.419079 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.419530 (+ 451us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.419706 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260927 02:50:49.420395 16644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43285
I20260927 02:50:49.420493 16644 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:49.420668 16644 heartbeater.cc:511] Master 127.0.98.62:43285 requested a full tablet report, sending...
I20260927 02:50:49.421041 16318 ts_manager.cc:194] Registered new tserver with Master: 862d02e84725424d9a6fdddfc26e41a3 (127.0.98.2:34325)
I20260927 02:50:49.421398 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34265
W20260927 02:50:49.498313 16647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:50:49.498519 16647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:50:49.498558 16647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:50:49.499729 16647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:50:49.499771 16647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:50:49.499796 16647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:50:49.501174 16647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43799
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:50:49.501379 16647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:50:49.501592 16647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:50:49.506351 16653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:49.506351 16652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:50:49.506470 16655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:50:49.506526 16647 server_base.cc:1034] running on GCE node
I20260927 02:50:49.506768 16647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:50:49.507025 16647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:50:49.508173 16647 hybrid_clock.cc:648] HybridClock initialized: now 1790477449508159 us; error 29 us; skew 500 ppm
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:50:49.510623 16647 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:50:49.511109 16647 webserver.cc:460] Webserver started at http://127.0.98.3:43799/ using document root <none> and password file <none>
I20260927 02:50:49.511299 16647 fs_manager.cc:359] Metadata directory not provided
I20260927 02:50:49.511348 16647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:50:49.512672 16647 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:50:49.513355 16662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.513540 16647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 02:50:49.513631 16647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "5c19b38dea454848b6e314eb1c7116d8"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "df95e816afe2ddbbdadf18548018e11c"
server_key_iv: "78e6333777422da50adebce26ca264c4"
server_key_version: "encryptionkey@0"
I20260927 02:50:49.513967 16647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:50:49.525920 16647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:50:49.528290 16647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:50:49.528426 16647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:50:49.528632 16647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:50:49.528901 16647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:50:49.528930 16647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.528951 16647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:50:49.528970 16647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.534578 16647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34325
I20260927 02:50:49.534657 16775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34325 every 8 connection(s)
I20260927 02:50:49.534991 16647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 02:50:49 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:50:49.544094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 16647
I20260927 02:50:49.544598 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.535813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49911 (local address 127.0.98.62:43285)
0927 02:50:49.535883 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.535885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.536066 (+ 181us) server_negotiation.cc:407] Connection header received
0927 02:50:49.536617 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.536618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.536628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.536690 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:49.537612 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.538205 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.539302 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.539566 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.539625 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.541822 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:49.541823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:49.541824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:49.541830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:49.543216 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.543611 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.543612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.543613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.543670 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.543888 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.543889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.543889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.544036 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:49.544104 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.544418 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.544508 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:50:49.544839 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.535753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:49911)
0927 02:50:49.535939 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.535979 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.536494 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.536746 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.536750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.537300 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:50:49.537505 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.537509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.538309 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.538311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.539211 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.539214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.539668 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.539669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.539784 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.540296 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:50:49.540305 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:50:49.541726 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:50:49.543277 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.543283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.543285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.543532 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.543717 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:50:49.543718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:50:49.543718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:50:49.543819 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:50:49.544126 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:50:49.544133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:50:49.544281 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.544383 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.544564 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260927 02:50:49.545403 16776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43285
I20260927 02:50:49.545519 16776 heartbeater.cc:464] Registering TS with master...
I20260927 02:50:49.545699 16776 heartbeater.cc:511] Master 127.0.98.62:43285 requested a full tablet report, sending...
I20260927 02:50:49.546078 16318 ts_manager.cc:194] Registered new tserver with Master: 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325)
I20260927 02:50:49.546578 16318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49911
I20260927 02:50:49.550691 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.544468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.98.62:43285)
0927 02:50:49.544614 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.544616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.544806 (+ 190us) server_negotiation.cc:407] Connection header received
0927 02:50:49.544825 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.544826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.544835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.544882 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:49.545518 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.546157 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.547181 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.547397 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.547466 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.548503 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:49.548504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:49.548504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:49.548510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:49.549805 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.550110 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.550111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.550112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.550163 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.550287 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.550287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.550288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.550378 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:49.550498 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.550563 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.550628 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260927 02:50:49.551331 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:50:49.553891 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.551667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.0.98.62:43285)
0927 02:50:49.551733 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.551749 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.551884 (+ 135us) server_negotiation.cc:407] Connection header received
0927 02:50:49.551918 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.551919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.551948 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.552014 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:50:49.552108 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.552548 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.553253 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.553436 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.553478 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.553644 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:50:49.553689 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 02:50:49.553834 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55056: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 02:50:49.553949 16780 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55056: token expired
W20260927 02:50:49.554013 16247 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:43285)
I20260927 02:50:49.559389 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.554228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.0.98.62:43285)
0927 02:50:49.554271 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.554273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.554281 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:49.554329 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.554330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.554339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.554393 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:50:49.554843 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.555290 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.555944 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.556142 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.556193 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.557261 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:50:49.557262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:50:49.557263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:50:49.557270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:50:49.558396 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.558708 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.558709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.558710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.558762 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:50:49.558939 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:50:49.558940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:50:49.558940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:50:49.559043 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:50:49.559126 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.559227 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.559300 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:50:49.560424 16318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:50:49.561972 16318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:50:49.564155 16440 tablet_service.cc:1467] Processing CreateTablet for tablet ff419227b6f842c18331c173f5038c32 (DEFAULT_TABLE table=test-table [id=c5a4364d7cf34b86ba8c65ca7ebaf78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:49.564358 16440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff419227b6f842c18331c173f5038c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:49.565788 16398 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap starting.
I20260927 02:50:49.566082 16398 tablet_bootstrap.cc:654] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:49.567401 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.563669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34325 (local address 127.0.0.1:45616)
0927 02:50:49.563860 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.563871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.563922 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.564118 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.564119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.564121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:49.564167 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.564169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.565200 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.565201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.566298 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.566301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.567132 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.567133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.567213 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.567237 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.567291 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.567334 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 02:50:49.567620 16398 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: No bootstrap required, opened a new log
I20260927 02:50:49.567672 16398 ts_tablet_manager.cc:1397] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 02:50:49.567662 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.563757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.0.98.3:34325)
0927 02:50:49.563941 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.563946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.563958 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:50:49.563992 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.563994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.564007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.564094 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:49.564231 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.565127 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.566366 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.567060 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.567114 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.567156 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.567336 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.567385 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":106,"threads_started":1}
I20260927 02:50:49.567827 16398 raft_consensus.cc:348] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.567903 16398 raft_consensus.cc:374] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:49.567947 16398 raft_consensus.cc:729] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf4e918d35a436dbf94391bc7d3191e, State: Initialized, Role: FOLLOWER
I20260927 02:50:49.567996 16398 consensus_queue.cc:260] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.568077 16398 ts_tablet_manager.cc:1428] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 02:50:49.568349 16710 tablet_service.cc:1467] Processing CreateTablet for tablet ff419227b6f842c18331c173f5038c32 (DEFAULT_TABLE table=test-table [id=c5a4364d7cf34b86ba8c65ca7ebaf78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:49.568682 16710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff419227b6f842c18331c173f5038c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:49.569409 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.563684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34325 (local address 127.0.0.1:56826)
0927 02:50:49.563915 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.563924 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.563953 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.566199 (+ 2246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.566200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.566203 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:49.566242 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.566244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.567324 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.567325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.568376 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.568378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.569176 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.569177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.569244 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.569247 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.569299 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.569341 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":112,"threads_started":1}
I20260927 02:50:49.569424 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.565830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.0.98.2:34325)
0927 02:50:49.566023 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.566027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.566038 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:50:49.566074 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.566076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.566090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.566178 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:49.566307 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.567206 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.568448 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.569106 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.569168 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.569220 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.569312 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.569349 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":92,"threads_started":1}
I20260927 02:50:49.570003 16578 tablet_service.cc:1467] Processing CreateTablet for tablet ff419227b6f842c18331c173f5038c32 (DEFAULT_TABLE table=test-table [id=c5a4364d7cf34b86ba8c65ca7ebaf78e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:50:49.570225 16578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff419227b6f842c18331c173f5038c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:50:49.571522 16786 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Bootstrap starting.
I20260927 02:50:49.572077 16786 tablet_bootstrap.cc:654] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:49.572539 16786 log.cc:824] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:49.572837 16788 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Bootstrap starting.
I20260927 02:50:49.573307 16786 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: No bootstrap required, opened a new log
I20260927 02:50:49.573366 16786 ts_tablet_manager.cc:1397] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:49.573391 16788 tablet_bootstrap.cc:654] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Neither blocks nor log segments found. Creating new log.
I20260927 02:50:49.573818 16788 log.cc:824] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:50:49.574476 16788 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: No bootstrap required, opened a new log
I20260927 02:50:49.574534 16788 ts_tablet_manager.cc:1397] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:50:49.574901 16786 raft_consensus.cc:348] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.575039 16786 raft_consensus.cc:374] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:49.575071 16786 raft_consensus.cc:729] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c19b38dea454848b6e314eb1c7116d8, State: Initialized, Role: FOLLOWER
I20260927 02:50:49.575194 16786 consensus_queue.cc:260] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.575446 16786 ts_tablet_manager.cc:1428] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:50:49.575474 16776 heartbeater.cc:503] Master 127.0.98.62:43285 was elected leader, sending a full tablet report...
I20260927 02:50:49.576108 16788 raft_consensus.cc:348] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.576256 16788 raft_consensus.cc:374] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:50:49.576278 16788 raft_consensus.cc:729] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862d02e84725424d9a6fdddfc26e41a3, State: Initialized, Role: FOLLOWER
I20260927 02:50:49.576386 16788 consensus_queue.cc:260] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.576624 16788 ts_tablet_manager.cc:1428] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:50:49.576654 16644 heartbeater.cc:503] Master 127.0.98.62:43285 was elected leader, sending a full tablet report...
W20260927 02:50:49.660863 16645 tablet.cc:2369] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:50:49.742218 16791 raft_consensus.cc:482] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:50:49.742349 16791 raft_consensus.cc:504] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.742781 16791 leader_election.cc:288] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325), aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709)
I20260927 02:50:49.746145 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.743091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59695 (local address 127.0.98.3:34325)
0927 02:50:49.743137 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.743139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.743241 (+ 102us) server_negotiation.cc:407] Connection header received
0927 02:50:49.743321 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.743322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.743355 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.743410 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:49.743541 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.744088 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.745354 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.745768 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.745828 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.745840 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.746035 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.746081 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:50:49.746165 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.743006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45709 (local address 127.0.98.2:47265)
0927 02:50:49.743067 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.743081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.743131 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.743329 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.743330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.743371 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:49.743464 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.743483 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.744216 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.744217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.745293 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.745295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.745916 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.745917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.745997 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.746000 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.746055 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.746090 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":75}
I20260927 02:50:49.746183 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.743022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34325 (local address 127.0.98.2:59695)
0927 02:50:49.743172 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.743182 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:50:49.743242 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:50:49.743416 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:50:49.743417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:50:49.743418 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:50:49.743478 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.743480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.744171 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.744172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.745270 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:50:49.745273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.745866 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.745867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:50:49.745932 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.745936 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:50:49.746023 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:50:49.746067 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":83,"thread_start_us":90,"threads_started":1}
I20260927 02:50:49.746495 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.743118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47265 (local address 127.0.98.1:45709)
0927 02:50:49.743176 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.743177 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.743187 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:50:49.743222 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.743223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.743251 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.743320 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:50:49.743589 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.744129 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.745379 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.745855 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.745917 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.745999 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.746393 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.746435 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:50:49.746534 16730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff419227b6f842c18331c173f5038c32" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c19b38dea454848b6e314eb1c7116d8" is_pre_election: true
I20260927 02:50:49.746740 16730 raft_consensus.cc:2454] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 0.
I20260927 02:50:49.746888 16460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff419227b6f842c18331c173f5038c32" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf4e918d35a436dbf94391bc7d3191e" is_pre_election: true
I20260927 02:50:49.746958 16532 leader_election.cc:302] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c19b38dea454848b6e314eb1c7116d8, 862d02e84725424d9a6fdddfc26e41a3; no voters:
I20260927 02:50:49.747052 16460 raft_consensus.cc:2454] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 0.
I20260927 02:50:49.747093 16791 raft_consensus.cc:2781] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:50:49.747160 16791 raft_consensus.cc:482] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:50:49.747191 16791 raft_consensus.cc:3037] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:49.747869 16791 raft_consensus.cc:504] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.748016 16791 leader_election.cc:288] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 election: Requested vote from peers 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325), aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709)
I20260927 02:50:49.748172 16730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff419227b6f842c18331c173f5038c32" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c19b38dea454848b6e314eb1c7116d8"
I20260927 02:50:49.748169 16460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff419227b6f842c18331c173f5038c32" candidate_uuid: "862d02e84725424d9a6fdddfc26e41a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf4e918d35a436dbf94391bc7d3191e"
I20260927 02:50:49.748241 16460 raft_consensus.cc:3037] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:49.748241 16730 raft_consensus.cc:3037] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:50:49.749029 16730 raft_consensus.cc:2454] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 1.
I20260927 02:50:49.749039 16460 raft_consensus.cc:2454] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862d02e84725424d9a6fdddfc26e41a3 in term 1.
I20260927 02:50:49.749189 16532 leader_election.cc:302] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c19b38dea454848b6e314eb1c7116d8, 862d02e84725424d9a6fdddfc26e41a3; no voters:
I20260927 02:50:49.749320 16791 raft_consensus.cc:2781] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:50:49.749454 16791 raft_consensus.cc:686] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 1 LEADER]: Becoming Leader. State: Replica: 862d02e84725424d9a6fdddfc26e41a3, State: Running, Role: LEADER
I20260927 02:50:49.749606 16791 consensus_queue.cc:237] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:50:49.750423 16317 catalog_manager.cc:5495] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 862d02e84725424d9a6fdddfc26e41a3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "862d02e84725424d9a6fdddfc26e41a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } health_report { overall_health: UNKNOWN } } }
W20260927 02:50:49.785554 16777 tablet.cc:2369] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:50:49.788170 16507 tablet.cc:2369] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:50:49.797294 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.794995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.0.98.2:34325)
0927 02:50:49.795045 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:50:49.795047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:50:49.795055 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:50:49.795130 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:50:49.795131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:50:49.795149 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:50:49.795238 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:50:49.795323 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.795801 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.796567 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:50:49.796758 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:50:49.796814 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:50:49.796978 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:50:49.797088 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:50:49.797164 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 02:50:49.797175 (+ 11us) server_negotiation.cc:299] Negotiation successful
0927 02:50:49.797193 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:50:49.799160 16730 raft_consensus.cc:1261] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Refusing update from remote peer 862d02e84725424d9a6fdddfc26e41a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:49.799178 16460 raft_consensus.cc:1261] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Refusing update from remote peer 862d02e84725424d9a6fdddfc26e41a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:50:49.799525 16791 consensus_queue.cc:1035] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:49.799664 16793 consensus_queue.cc:1035] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:50:49.801810 16796 mvcc.cc:205] Tried to move back new op lower bound from 7333795634374737920 to 7333795634174939136. Current Snapshot: MvccSnapshot[applied={T|T < 7333795634374737920}]
I20260927 02:50:49.802457 16797 mvcc.cc:205] Tried to move back new op lower bound from 7333795634374737920 to 7333795634174939136. Current Snapshot: MvccSnapshot[applied={T|T < 7333795634374737920}]
I20260927 02:50:49.804085 16317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:49.804170 16317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:50:49.804826 16317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Sending DeleteTablet for 3 replicas of tablet ff419227b6f842c18331c173f5038c32
I20260927 02:50:49.805169 16440 tablet_service.cc:1514] Processing DeleteTablet for tablet ff419227b6f842c18331c173f5038c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59572
I20260927 02:50:49.805160 16710 tablet_service.cc:1514] Processing DeleteTablet for tablet ff419227b6f842c18331c173f5038c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45616
I20260927 02:50:49.805186 16578 tablet_service.cc:1514] Processing DeleteTablet for tablet ff419227b6f842c18331c173f5038c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:50:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56826
I20260927 02:50:49.805342 16515 tablet_replica.cc:330] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: stopping tablet replica
I20260927 02:50:49.805406 16515 raft_consensus.cc:2229] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:49.805455 16515 raft_consensus.cc:2258] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:50:49.805634 16804 tablet_replica.cc:330] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: stopping tablet replica
I20260927 02:50:49.805627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 16356
W20260927 02:50:49.809706 16305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45709 (error 108)
I20260927 02:50:49.810480 16804 raft_consensus.cc:2229] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:50:49.810573 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.810432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45709 (local address 127.0.0.1:59586)
0927 02:50:49.810487 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.810537 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 02:50:49.810617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 16512
I20260927 02:50:49.810698 16804 raft_consensus.cc:2258] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:50:49.810834 16305 catalog_manager.cc:4584] TS aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff419227b6f842c18331c173f5038c32: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45709: connect: Connection refused (error 111)
I20260927 02:50:49.811868 16804 ts_tablet_manager.cc:1905] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:50:49.813740 16804 ts_tablet_manager.cc:1918] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:50:49.813827 16804 log.cc:1196] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/ff419227b6f842c18331c173f5038c32
I20260927 02:50:49.814074 16804 ts_tablet_manager.cc:1939] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Deleting consensus metadata
I20260927 02:50:49.814483 16303 catalog_manager.cc:4841] TS 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325): tablet ff419227b6f842c18331c173f5038c32 (table test-table [id=c5a4364d7cf34b86ba8c65ca7ebaf78e]) successfully deleted
I20260927 02:50:49.815327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 16647
I20260927 02:50:49.815404 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:50:49.815302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34325 (local address 127.0.0.1:56834)
0927 02:50:49.815346 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:50:49.815371 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:50:49.822006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 16285
I20260927 02:51:09.826649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43285
--webserver_interface=127.0.98.62
--webserver_port=46405
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:09.904784 16807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:09.904992 16807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:09.905022 16807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:09.906158 16807 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 02:51:09.906198 16807 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 02:51:09.906213 16807 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 02:51:09.906229 16807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:09.906244 16807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:09.906262 16807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:09.906284 16807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:09.906301 16807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:09.907832 16807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:09.908070 16807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:09.908278 16807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:51:09.913146 16807 server_base.cc:1034] running on GCE node
W20260927 02:51:09.913113 16812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:09.913125 16815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:09.913107 16813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:09.913568 16807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:09.913869 16807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:09.915001 16807 hybrid_clock.cc:648] HybridClock initialized: now 1790477469914988 us; error 30 us; skew 500 ppm
Sep 27 02:51:09 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:09.917282 16807 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:09.917701 16807 webserver.cc:460] Webserver started at http://127.0.98.62:46405/ using document root <none> and password file <none>
I20260927 02:51:09.917891 16807 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:09.917937 16807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:09.919262 16807 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:09.919906 16822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:09.920045 16807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:09.920125 16807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "a246b664d53b4b74a4d889e55940dc9d"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "fe397ae837d33c38892d355bbbeea355"
server_key_iv: "69aeaae1404221603e9b2e2894449078"
server_key_version: "encryptionkey@0"
I20260927 02:51:09.920454 16807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:09.933012 16807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:09.934770 16807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:09.934947 16807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:09.938521 16807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43285
I20260927 02:51:09.938562 16874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43285 every 8 connection(s)
I20260927 02:51:09.938947 16807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:09.940336 16875 sys_catalog.cc:263] Verifying existing consensus state
I20260927 02:51:09.940986 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Bootstrap starting.
I20260927 02:51:09.941129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 16807
I20260927 02:51:09.941426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45709
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43285
--webserver_port=37661
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:51:09.949587 16875 log.cc:824] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:09.951790 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:51:09.952045 16875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: Bootstrap complete.
I20260927 02:51:09.954017 16875 raft_consensus.cc:348] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:51:09.954234 16875 raft_consensus.cc:729] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a246b664d53b4b74a4d889e55940dc9d, State: Initialized, Role: FOLLOWER
I20260927 02:51:09.954376 16875 consensus_queue.cc:260] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:51:09.954447 16875 raft_consensus.cc:388] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:09.954494 16875 raft_consensus.cc:482] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:09.954540 16875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 2 FOLLOWER]: Advancing to term 3
I20260927 02:51:09.955560 16875 raft_consensus.cc:504] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:51:09.955664 16875 leader_election.cc:302] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a246b664d53b4b74a4d889e55940dc9d; no voters:
I20260927 02:51:09.955821 16875 leader_election.cc:288] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 02:51:09.955870 16879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 02:51:09.955982 16879 raft_consensus.cc:686] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [term 3 LEADER]: Becoming Leader. State: Replica: a246b664d53b4b74a4d889e55940dc9d, State: Running, Role: LEADER
I20260927 02:51:09.956077 16875 sys_catalog.cc:564] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:09.956074 16879 consensus_queue.cc:237] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } }
I20260927 02:51:09.956377 16881 sys_catalog.cc:455] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a246b664d53b4b74a4d889e55940dc9d. Latest consensus state: current_term: 3 leader_uuid: "a246b664d53b4b74a4d889e55940dc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } } }
I20260927 02:51:09.956385 16880 sys_catalog.cc:455] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a246b664d53b4b74a4d889e55940dc9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a246b664d53b4b74a4d889e55940dc9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43285 } } }
I20260927 02:51:09.956470 16881 sys_catalog.cc:458] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:09.956476 16880 sys_catalog.cc:458] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:09.957748 16895 catalog_manager.cc:1259] Loaded cluster ID: e03d5dd9ca9b4a138744f70a1881a22b
I20260927 02:51:09.957795 16895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: loading cluster ID for follower catalog manager: success
I20260927 02:51:09.959021 16895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a246b664d53b4b74a4d889e55940dc9d: acquiring CA information for follower catalog manager: success
F20260927 02:51:09.959399 16895 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f34e56bc9ad google::LogMessage::Fail() at ??:0
@ 0x7f34e56bf88a google::LogMessage::SendToLog() at ??:0
@ 0x7f34e56bc4de google::LogMessage::Flush() at ??:0
@ 0x7f34e56bda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f34e7e29104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f34ed6196b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f34ed619989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f34ed619e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f34ed61c8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f34e757f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f34e83e36db start_thread at ??:0
@ 0x7f34e41ba61f clone at ??:0
W20260927 02:51:10.033686 16877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:10.033890 16877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:10.033911 16877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:10.035081 16877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:10.035120 16877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:10.035145 16877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:51:10.036576 16877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37661
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:10.036787 16877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:10.036995 16877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:10.041659 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:10.041646 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:10.041649 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:10.041910 16877 server_base.cc:1034] running on GCE node
I20260927 02:51:10.042126 16877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:10.042347 16877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:10.043529 16877 hybrid_clock.cc:648] HybridClock initialized: now 1790477470043515 us; error 28 us; skew 500 ppm
Sep 27 02:51:10 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:10.046031 16877 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:51:10.046458 16877 webserver.cc:460] Webserver started at http://127.0.98.1:37661/ using document root <none> and password file <none>
I20260927 02:51:10.046653 16877 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:10.046698 16877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:10.048017 16877 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:10.048616 16923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:10.048813 16877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:10.048914 16877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "aaf4e918d35a436dbf94391bc7d3191e"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "5f912379f9ff877673705fa34bc2a709"
server_key_iv: "6911cfe66840fec9330b05dc4a9bc6fa"
server_key_version: "encryptionkey@0"
I20260927 02:51:10.049358 16877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:10.065419 16877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:10.067945 16877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:10.068102 16877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:10.068357 16877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:10.068862 16930 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:51:10.070330 16877 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:51:10.070391 16877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:51:10.070425 16877 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:51:10.071326 16877 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:51:10.071369 16877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:10.071476 16930 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap starting.
I20260927 02:51:10.077765 16877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45709
I20260927 02:51:10.078241 16877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:51:10.081645 16930 log.cc:824] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:10.082712 17037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45709 every 8 connection(s)
I20260927 02:51:10.084069 16930 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:51:10.084313 16930 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Bootstrap complete.
I20260927 02:51:10.085031 16930 ts_tablet_manager.cc:1397] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260927 02:51:10.086027 16930 raft_consensus.cc:348] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:10.086257 16930 raft_consensus.cc:729] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf4e918d35a436dbf94391bc7d3191e, State: Initialized, Role: FOLLOWER
I20260927 02:51:10.086426 16930 consensus_queue.cc:260] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:10.086479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 16877
I20260927 02:51:10.086732 16930 ts_tablet_manager.cc:1428] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260927 02:51:10.086740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34325
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43285
--webserver_port=45507
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:10.142006 17040 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:10.079079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:50185)
0927 02:51:10.079266 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.079312 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:10.079765 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:10.141836 (+ 62071us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
W20260927 02:51:10.142259 17038 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43285 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43285: Client connection negotiation failed: client connection to 127.0.98.62:43285: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 02:51:10.142552 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.142456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:34765)
0927 02:51:10.142500 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.142534 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 02:51:10.142887 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.142773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:54569)
0927 02:51:10.142840 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.142863 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 02:51:10.142992 17038 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 02:51:10.167414 17043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:10.167639 17043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:10.167670 17043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:10.168792 17043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:10.168833 17043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:10.168867 17043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:51:10.170303 17043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45507
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:10.170526 17043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:10.170742 17043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:51:10.175487 17043 server_base.cc:1034] running on GCE node
W20260927 02:51:10.175455 17049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:10.175534 17048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:10.175455 17051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:10.175776 17043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:10.175974 17043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:10.177111 17043 hybrid_clock.cc:648] HybridClock initialized: now 1790477470177084 us; error 39 us; skew 500 ppm
Sep 27 02:51:10 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:10.179490 17043 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:51:10.179868 17043 webserver.cc:460] Webserver started at http://127.0.98.2:45507/ using document root <none> and password file <none>
I20260927 02:51:10.180048 17043 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:10.180090 17043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:10.181313 17043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:10.181887 17058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:10.182041 17043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:10.182127 17043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "862d02e84725424d9a6fdddfc26e41a3"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "409025fbe4c270a5e40357bd34526f8f"
server_key_iv: "6b56353a064bb303344912535f84bef1"
server_key_version: "encryptionkey@0"
I20260927 02:51:10.182441 17043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:10.201279 17043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:10.203423 17043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:10.203533 17043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:10.203729 17043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:10.204079 17065 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 02:51:10.205463 17043 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 02:51:10.205508 17043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 02:51:10.205531 17043 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 02:51:10.206172 17043 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 02:51:10.206203 17043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 02:51:10.206269 17065 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Bootstrap starting.
I20260927 02:51:10.211521 17043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34325
I20260927 02:51:10.211663 17172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34325 every 8 connection(s)
I20260927 02:51:10.211910 17043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:51:10.213013 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.212547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:39067)
0927 02:51:10.212716 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.212815 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
W20260927 02:51:10.213232 17173 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43285 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43285: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
I20260927 02:51:10.213495 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.213400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:55719)
0927 02:51:10.213441 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.213472 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:51:10.213774 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.213683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:52043)
0927 02:51:10.213726 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.213752 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:51:10.213876 17173 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:51:10.215989 17065 log.cc:824] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:10.218117 17065 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:51:10.218331 17065 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Bootstrap complete.
I20260927 02:51:10.219105 17065 ts_tablet_manager.cc:1397] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260927 02:51:10.220223 17065 raft_consensus.cc:348] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:10.220466 17065 raft_consensus.cc:729] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862d02e84725424d9a6fdddfc26e41a3, State: Initialized, Role: FOLLOWER
I20260927 02:51:10.220574 17065 consensus_queue.cc:260] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:10.220767 17065 ts_tablet_manager.cc:1428] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:51:10.221024 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.220931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:37243)
0927 02:51:10.220969 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.221004 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:51:10.221334 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.221255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:60203)
0927 02:51:10.221293 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.221316 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 02:51:10.221343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 17043
I20260927 02:51:10.221580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34325
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43285
--webserver_port=43799
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:10.300155 17178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:10.300359 17178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:10.300390 17178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:10.301545 17178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:10.301589 17178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:10.301623 17178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:51:10.303073 17178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34325
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43799
--tserver_master_addrs=127.0.98.62:43285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:10.303290 17178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:10.303490 17178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:10.308133 17183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:10.308147 17186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:10.308154 17184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:10.308378 17178 server_base.cc:1034] running on GCE node
I20260927 02:51:10.308595 17178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:10.308801 17178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:10.309942 17178 hybrid_clock.cc:648] HybridClock initialized: now 1790477470309929 us; error 28 us; skew 500 ppm
Sep 27 02:51:10 dist-test-slave-f4rg krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:10.312256 17178 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:51:10.312701 17178 webserver.cc:460] Webserver started at http://127.0.98.3:43799/ using document root <none> and password file <none>
I20260927 02:51:10.312888 17178 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:10.312935 17178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:10.314148 17178 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:10.314675 17193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:10.314846 17178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:10.314934 17178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "5c19b38dea454848b6e314eb1c7116d8"
format_stamp: "Formatted at 2026-09-27 02:50:28 on dist-test-slave-f4rg"
server_key: "df95e816afe2ddbbdadf18548018e11c"
server_key_iv: "78e6333777422da50adebce26ca264c4"
server_key_version: "encryptionkey@0"
I20260927 02:51:10.315248 17178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:10.340443 17178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:10.342376 17178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:10.342518 17178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:10.342732 17178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:10.343039 17178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:10.343071 17178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:10.343101 17178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:10.343122 17178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:10.348565 17178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34325
I20260927 02:51:10.348620 17306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34325 every 8 connection(s)
I20260927 02:51:10.348968 17178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:51:10.350100 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.349709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:52837)
0927 02:51:10.349865 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.349940 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260927 02:51:10.350517 17307 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43285 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43285: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
I20260927 02:51:10.350776 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.350700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:50655)
0927 02:51:10.350736 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.350760 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 02:51:10.351034 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:10.350948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:48647)
0927 02:51:10.350981 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:10.351000 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:51:10.351132 17307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 02:51:10.356235 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 17178
I20260927 02:51:11.143572 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.143338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:43527)
0927 02:51:11.143490 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:11.143525 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
W20260927 02:51:11.212936 17168 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.2:34325
I20260927 02:51:11.222035 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.221757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:51293)
0927 02:51:11.221905 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:11.221949 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
W20260927 02:51:11.349571 17302 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.3:34325
I20260927 02:51:11.351610 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.351400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:45251)
0927 02:51:11.351518 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:11.351560 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260927 02:51:11.495092 17314 raft_consensus.cc:482] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:51:11.495216 17314 raft_consensus.cc:504] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:11.495548 17314 leader_election.cc:288] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325), 862d02e84725424d9a6fdddfc26e41a3 (127.0.98.2:34325)
Sep 27 02:51:11 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 02:51:11 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 02:51:11.509065 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.495882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40809 (local address 127.0.98.2:34325)
0927 02:51:11.495986 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:11.495997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:11.496020 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:51:11.496121 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:11.496123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:11.496164 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:11.496269 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:11.497110 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.497973 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.498989 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.499393 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.499547 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:11.505590 (+ 6043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:11.505594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:11.505595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:11.505606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:11.507315 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:11.507782 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:11.507785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:11.507785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:11.507898 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:11.508186 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:11.508187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:11.508187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:11.508487 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:11.508546 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:11.508842 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 02:51:11.508963 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":46,"threads_started":1}
I20260927 02:51:11.509068 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.495742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34325 (local address 127.0.98.1:40809)
0927 02:51:11.495787 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:11.495798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:11.495857 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:11.496332 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:11.496335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:11.496791 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:11.497000 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:11.497006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.498071 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.498072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:11.498914 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:11.498916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.499472 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.499473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:11.499584 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:11.503977 (+ 4393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:11.503987 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:11.505469 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:11.507396 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:11.507403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:11.507405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:11.507691 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:11.507978 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:11.507979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:11.507979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:11.508098 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:11.508551 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:11.508558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:11.508698 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 02:51:11.508779 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:51:11.508970 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":48}
I20260927 02:51:11.509083 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.495825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33277 (local address 127.0.98.3:34325)
0927 02:51:11.495973 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:11.495987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:11.496020 (+ 33us) server_negotiation.cc:407] Connection header received
0927 02:51:11.496556 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:11.496559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:11.496607 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:11.496748 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:11.497109 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.497975 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.499000 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.499391 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.499485 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:11.505987 (+ 6502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:11.505991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:11.505991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:11.506002 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:11.507569 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:11.507939 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:11.507942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:11.507943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:11.508060 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:11.508322 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:11.508322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:11.508323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:11.508550 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:11.508628 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:11.508845 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 02:51:11.508963 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260927 02:51:11.509150 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.495742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34325 (local address 127.0.98.1:33277)
0927 02:51:11.495936 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:11.495944 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:11.495979 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:11.496732 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:11.496733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:11.496890 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:11.497000 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:11.497005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.498075 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.498076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:11.498914 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:11.498916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:11.499471 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:11.499472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:11.499584 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:11.503983 (+ 4399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:11.503988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:11.505886 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:11.507656 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:11.507657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:11.507657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:11.507836 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:11.508162 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:11.508163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:11.508164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:11.508247 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:11.508623 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:11.508623 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:11.508693 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 02:51:11.508779 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:51:11.509045 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":225,"thread_start_us":60,"threads_started":1}
W20260927 02:51:11.509850 16925 leader_election.cc:341] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325): Not found: Tablet not found: ff419227b6f842c18331c173f5038c32
I20260927 02:51:11.510251 17127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff419227b6f842c18331c173f5038c32" candidate_uuid: "aaf4e918d35a436dbf94391bc7d3191e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "862d02e84725424d9a6fdddfc26e41a3" is_pre_election: true
I20260927 02:51:11.510385 17127 raft_consensus.cc:2454] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaf4e918d35a436dbf94391bc7d3191e in term 1.
I20260927 02:51:11.510609 16926 leader_election.cc:302] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862d02e84725424d9a6fdddfc26e41a3, aaf4e918d35a436dbf94391bc7d3191e; no voters: 5c19b38dea454848b6e314eb1c7116d8
I20260927 02:51:11.510715 17314 raft_consensus.cc:2781] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 02:51:11.510788 17314 raft_consensus.cc:482] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:51:11.510831 17314 raft_consensus.cc:3037] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:51:11.511924 17314 raft_consensus.cc:504] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:11.512046 17314 leader_election.cc:288] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [CANDIDATE]: Term 2 election: Requested vote from peers 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325), 862d02e84725424d9a6fdddfc26e41a3 (127.0.98.2:34325)
I20260927 02:51:11.512236 17127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff419227b6f842c18331c173f5038c32" candidate_uuid: "aaf4e918d35a436dbf94391bc7d3191e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "862d02e84725424d9a6fdddfc26e41a3"
W20260927 02:51:11.512295 16925 leader_election.cc:341] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325): Not found: Tablet not found: ff419227b6f842c18331c173f5038c32
I20260927 02:51:11.512320 17127 raft_consensus.cc:3037] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 1 FOLLOWER]: Advancing to term 2
I20260927 02:51:11.513303 17127 raft_consensus.cc:2454] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaf4e918d35a436dbf94391bc7d3191e in term 2.
I20260927 02:51:11.513440 16926 leader_election.cc:302] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 862d02e84725424d9a6fdddfc26e41a3, aaf4e918d35a436dbf94391bc7d3191e; no voters: 5c19b38dea454848b6e314eb1c7116d8
I20260927 02:51:11.513527 17314 raft_consensus.cc:2781] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 02:51:11.513639 17314 raft_consensus.cc:686] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [term 2 LEADER]: Becoming Leader. State: Replica: aaf4e918d35a436dbf94391bc7d3191e, State: Running, Role: LEADER
I20260927 02:51:11.513712 17314 consensus_queue.cc:237] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:11.513926 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:11.513822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:42711)
0927 02:51:11.513855 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:11.513898 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 02:51:12.024163 16925 consensus_peers.cc:487] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e -> Peer 5c19b38dea454848b6e314eb1c7116d8 (127.0.98.3:34325): Couldn't send request to peer 5c19b38dea454848b6e314eb1c7116d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff419227b6f842c18331c173f5038c32. This is attempt 1: this message will repeat every 5th retry.
I20260927 02:51:12.061254 17127 raft_consensus.cc:1261] T ff419227b6f842c18331c173f5038c32 P 862d02e84725424d9a6fdddfc26e41a3 [term 2 FOLLOWER]: Refusing update from remote peer aaf4e918d35a436dbf94391bc7d3191e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 02:51:12.061632 17320 consensus_queue.cc:1035] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e [LEADER]: Connected to new peer: Peer: permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 02:51:12.062515 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.062321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:39995)
0927 02:51:12.062414 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:12.062472 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
I20260927 02:51:12.063354 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.063224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:32823)
0927 02:51:12.063293 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:12.063323 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55,"thread_start_us":37,"threads_started":1}
I20260927 02:51:12.063369 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.063288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:50633)
0927 02:51:12.063316 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:12.063341 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260927 02:51:12.352212 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.351981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:58893)
0927 02:51:12.352125 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:12.352163 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260927 02:51:12.572682 17329 ts_tablet_manager.cc:927] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Initiating tablet copy from peer aaf4e918d35a436dbf94391bc7d3191e (127.0.98.1:45709)
I20260927 02:51:12.572916 17329 tablet_copy_client.cc:316] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:45709
Sep 27 02:51:12 dist-test-slave-f4rg krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 02:51:12.580799 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.573177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45709 (local address 127.0.98.3:43721)
0927 02:51:12.573246 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:12.573267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:12.573342 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:12.573546 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:12.573553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:12.573841 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:12.573900 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:12.573902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:12.574731 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:12.574731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:12.575364 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:12.575367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:12.575979 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:12.575980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:12.576049 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:12.576541 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:12.576545 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:12.577945 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:12.579671 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:12.579678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:12.579681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:12.579876 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:12.580077 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:12.580078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:12.580078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:12.580137 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:12.580425 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:12.580433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:12.580489 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 02:51:12.580575 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:51:12.580698 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 02:51:12.580816 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.573300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43721 (local address 127.0.98.1:45709)
0927 02:51:12.573426 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:12.573430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:12.573438 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:12.573460 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:12.573462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:12.573473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:12.573522 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:12.573965 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:12.574652 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:12.575461 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:12.575809 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:12.575879 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:12.578039 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:12.578048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:12.578048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:12.578056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:12.579590 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:12.579953 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:12.579956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:12.579957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:12.580013 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:12.580188 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:12.580189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:12.580189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:12.580357 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:12.580426 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:12.580657 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 02:51:12.580745 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":55,"threads_started":1}
I20260927 02:51:12.581225 17011 tablet_copy_service.cc:140] P aaf4e918d35a436dbf94391bc7d3191e: Received BeginTabletCopySession request for tablet ff419227b6f842c18331c173f5038c32 from peer 5c19b38dea454848b6e314eb1c7116d8 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43721)
I20260927 02:51:12.581290 17011 tablet_copy_service.cc:161] P aaf4e918d35a436dbf94391bc7d3191e: Beginning new tablet copy session on tablet ff419227b6f842c18331c173f5038c32 from peer 5c19b38dea454848b6e314eb1c7116d8 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43721: session id = 5c19b38dea454848b6e314eb1c7116d8-ff419227b6f842c18331c173f5038c32
I20260927 02:51:12.581743 17011 tablet_copy_source_session.cc:205] T ff419227b6f842c18331c173f5038c32 P aaf4e918d35a436dbf94391bc7d3191e: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 02:51:12.582648 17329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff419227b6f842c18331c173f5038c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:12.585260 17329 tablet_copy_client.cc:791] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: tablet copy: Starting download of 0 data blocks...
I20260927 02:51:12.585391 17329 tablet_copy_client.cc:655] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: tablet copy: Starting download of 1 WAL segments...
I20260927 02:51:12.586243 17329 tablet_copy_client.cc:531] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 02:51:12.586998 17329 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Bootstrap starting.
I20260927 02:51:12.588370 17329 log.cc:824] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:12.590631 17329 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 02:51:12.590901 17329 tablet_bootstrap.cc:492] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Bootstrap complete.
I20260927 02:51:12.591034 17329 ts_tablet_manager.cc:1397] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260927 02:51:12.591996 17329 raft_consensus.cc:348] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:12.592149 17329 raft_consensus.cc:729] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c19b38dea454848b6e314eb1c7116d8, State: Initialized, Role: FOLLOWER
I20260927 02:51:12.592267 17329 consensus_queue.cc:260] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c19b38dea454848b6e314eb1c7116d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34325 } } peers { permanent_uuid: "862d02e84725424d9a6fdddfc26e41a3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34325 } } peers { permanent_uuid: "aaf4e918d35a436dbf94391bc7d3191e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45709 } }
I20260927 02:51:12.592504 17329 ts_tablet_manager.cc:1428] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:12.592828 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.592718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:52577)
0927 02:51:12.592764 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:12.592793 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:51:12.592941 17011 tablet_copy_service.cc:342] P aaf4e918d35a436dbf94391bc7d3191e: Request end of tablet copy session 5c19b38dea454848b6e314eb1c7116d8-ff419227b6f842c18331c173f5038c32 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43721
I20260927 02:51:12.593001 17011 tablet_copy_service.cc:433] P aaf4e918d35a436dbf94391bc7d3191e: ending tablet copy session 5c19b38dea454848b6e314eb1c7116d8-ff419227b6f842c18331c173f5038c32 on tablet ff419227b6f842c18331c173f5038c32 with peer 5c19b38dea454848b6e314eb1c7116d8
I20260927 02:51:12.593089 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:12.593016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:42211)
0927 02:51:12.593048 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:12.593067 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 02:51:13.063921 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:13.063735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:35157)
0927 02:51:13.063844 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:13.063878 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260927 02:51:13.063921 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:13.063741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:43317)
0927 02:51:13.063853 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:13.063880 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":53,"threads_started":1}
I20260927 02:51:13.125052 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:13.124929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:54673)
0927 02:51:13.124979 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:13.125011 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:51:13.125080 17261 raft_consensus.cc:1203] T ff419227b6f842c18331c173f5038c32 P 5c19b38dea454848b6e314eb1c7116d8 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 02:51:13.125619 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:13.125448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:37467)
0927 02:51:13.125532 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:13.125563 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":39,"threads_started":1}
I20260927 02:51:14.064518 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:14.064296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:37737)
0927 02:51:14.064426 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:14.064469 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260927 02:51:14.125635 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:14.125435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:56375)
0927 02:51:14.125546 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:14.125580 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260927 02:51:14.126125 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:14.125936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:43321)
0927 02:51:14.126061 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:14.126088 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260927 02:51:15.065249 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:15.065008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:41405)
0927 02:51:15.065156 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:15.065201 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260927 02:51:15.126296 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:15.126085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:59267)
0927 02:51:15.126215 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:15.126250 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260927 02:51:15.126667 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:15.126550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:42099)
0927 02:51:15.126600 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:15.126636 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:51:15.126688 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:15.126506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:39005)
0927 02:51:15.126622 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:15.126651 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":46,"threads_started":1}
I20260927 02:51:16.066080 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:16.065847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:40733)
0927 02:51:16.065984 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:16.066032 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260927 02:51:16.127254 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:16.127044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:42935)
0927 02:51:16.127174 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:16.127207 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260927 02:51:16.127329 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:16.127124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:49179)
0927 02:51:16.127256 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:16.127287 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260927 02:51:17.066696 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:17.066477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:60261)
0927 02:51:17.066614 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:17.066650 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260927 02:51:17.127883 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:17.127683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:56477)
0927 02:51:17.127806 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:17.127838 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260927 02:51:17.127905 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:17.127700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:33199)
0927 02:51:17.127834 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:17.127865 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260927 02:51:18.067555 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:18.067263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:45489)
0927 02:51:18.067409 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:18.067507 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260927 02:51:18.128542 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:18.128343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:60911)
0927 02:51:18.128459 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:18.128491 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260927 02:51:18.128578 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:18.128331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:45223)
0927 02:51:18.128470 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:18.128497 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260927 02:51:19.068279 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:19.068017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:58869)
0927 02:51:19.068170 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:19.068230 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260927 02:51:19.129189 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:19.128991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:54945)
0927 02:51:19.129121 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:19.129147 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260927 02:51:19.129189 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:19.128989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:35711)
0927 02:51:19.129108 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:19.129140 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260927 02:51:20.069026 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:20.068783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:43745)
0927 02:51:20.068911 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:20.068968 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260927 02:51:20.129920 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:20.129683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:44297)
0927 02:51:20.129833 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:20.129876 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260927 02:51:20.129935 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:20.129683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:43769)
0927 02:51:20.129863 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:20.129894 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":94,"threads_started":1}
I20260927 02:51:21.069703 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:21.069471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:35901)
0927 02:51:21.069605 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:21.069644 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260927 02:51:21.130534 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:21.130311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:47679)
0927 02:51:21.130447 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:21.130478 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260927 02:51:21.130546 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:21.130360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:35933)
0927 02:51:21.130466 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:21.130507 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260927 02:51:22.070417 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:22.070170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:53213)
0927 02:51:22.070325 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:22.070370 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260927 02:51:22.131206 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:22.130992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:58453)
0927 02:51:22.131108 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:22.131143 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260927 02:51:22.131206 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:22.130989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:41785)
0927 02:51:22.131128 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:22.131159 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260927 02:51:23.071166 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:23.070937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:48999)
0927 02:51:23.071056 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:23.071118 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260927 02:51:23.131893 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:23.131676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:47465)
0927 02:51:23.131806 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:23.131846 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260927 02:51:23.131904 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:23.131676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:40091)
0927 02:51:23.131831 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:23.131864 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260927 02:51:24.071846 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:24.071644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:43555)
0927 02:51:24.071762 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:24.071797 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260927 02:51:24.132575 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:24.132354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:51053)
0927 02:51:24.132486 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:24.132529 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260927 02:51:24.132587 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:24.132354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:58067)
0927 02:51:24.132512 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:24.132546 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260927 02:51:25.072527 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:25.072279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.2:35325)
0927 02:51:25.072421 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:25.072468 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":107,"threads_started":1}
I20260927 02:51:25.133225 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:25.133000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.3:45645)
0927 02:51:25.133140 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:25.133178 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260927 02:51:25.133240 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:25.133060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43285 (local address 127.0.98.1:51605)
0927 02:51:25.133167 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:25.133200 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260927 02:51:25.363903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 16877
I20260927 02:51:25.368510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 17043
I20260927 02:51:25.372422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 17178
2026-09-27T02:51:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58203 ms)
[----------] 30 tests from SecurityITest (253238 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:25 dist-test-slave-f4rg krb5kdc[17378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:25 dist-test-slave-f4rg krb5kdc[17378](info): set up 2 sockets
Sep 27 02:51:25 dist-test-slave-f4rg krb5kdc[17378](info): commencing operation
krb5kdc: starting...
W20260927 02:51:26.318257 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:26.427567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39513
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:26.506580 17391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:26.506786 17391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:26.506829 17391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:26.507987 17391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:26.508030 17391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:26.508044 17391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:26.508062 17391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:26.508073 17391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:26.509493 17391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:26.509683 17391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:26.509887 17391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:51:26.514583 17391 server_base.cc:1034] running on GCE node
W20260927 02:51:26.514549 17397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:26.514552 17396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:26.514676 17399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:26.515193 17391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:26.515491 17391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:26.516623 17391 hybrid_clock.cc:648] HybridClock initialized: now 1790477486516614 us; error 28 us; skew 500 ppm
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:26.518951 17391 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:26.519326 17391 webserver.cc:460] Webserver started at http://127.0.98.62:38517/ using document root <none> and password file <none>
I20260927 02:51:26.519502 17391 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:26.519541 17391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:26.519630 17391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:26.520612 17391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "28983ff0217640b7bad6d7b533c51476"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "b63c480f484bbfdef89476ea6043459d"
server_key_iv: "fa2c3f9f3a84f88c11b64ff0b7a044c6"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.520907 17391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "28983ff0217640b7bad6d7b533c51476"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "b63c480f484bbfdef89476ea6043459d"
server_key_iv: "fa2c3f9f3a84f88c11b64ff0b7a044c6"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.522037 17391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:26.522625 17406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.522775 17391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:26.522892 17391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "28983ff0217640b7bad6d7b533c51476"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "b63c480f484bbfdef89476ea6043459d"
server_key_iv: "fa2c3f9f3a84f88c11b64ff0b7a044c6"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.522953 17391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:26.534288 17391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:26.536356 17391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:26.536487 17391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:26.540092 17391 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39513
I20260927 02:51:26.540135 17458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39513 every 8 connection(s)
I20260927 02:51:26.540433 17391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:26.541224 17459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:26.541805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 17391
I20260927 02:51:26.541894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:26.542062 392 external_mini_cluster.cc:1428] Setting key 9c166225626195f4d2be5cc04a696fb7
I20260927 02:51:26.543994 17459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476: Bootstrap starting.
I20260927 02:51:26.544554 17459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:26.544801 17459 log.cc:824] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:26.545368 17459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476: No bootstrap required, opened a new log
I20260927 02:51:26.547102 17459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28983ff0217640b7bad6d7b533c51476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260927 02:51:26.547230 17459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:26.547260 17459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28983ff0217640b7bad6d7b533c51476, State: Initialized, Role: FOLLOWER
I20260927 02:51:26.547379 17459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28983ff0217640b7bad6d7b533c51476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260927 02:51:26.547477 17459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:26.547515 17459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:26.547561 17459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:26.548038 17459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28983ff0217640b7bad6d7b533c51476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260927 02:51:26.548136 17459 leader_election.cc:302] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28983ff0217640b7bad6d7b533c51476; no voters:
I20260927 02:51:26.548283 17459 leader_election.cc:288] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:26.548310 17464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:26.548417 17464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [term 1 LEADER]: Becoming Leader. State: Replica: 28983ff0217640b7bad6d7b533c51476, State: Running, Role: LEADER
I20260927 02:51:26.548534 17459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:26.548548 17464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28983ff0217640b7bad6d7b533c51476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } }
I20260927 02:51:26.548933 17465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28983ff0217640b7bad6d7b533c51476. Latest consensus state: current_term: 1 leader_uuid: "28983ff0217640b7bad6d7b533c51476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28983ff0217640b7bad6d7b533c51476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } } }
I20260927 02:51:26.549053 17466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28983ff0217640b7bad6d7b533c51476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28983ff0217640b7bad6d7b533c51476" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39513 } } }
I20260927 02:51:26.549113 17465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:26.549188 17466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:26.549803 17468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:26.550319 17468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:26.551875 17468 catalog_manager.cc:1347] Generated new cluster ID: c75f1645a6a44bb692afb9d6ada72138
I20260927 02:51:26.551920 17468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:26.552207 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:26.542518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.0.98.62:39513)
0927 02:51:26.542667 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:26.542677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:26.542702 (+ 25us) server_negotiation.cc:407] Connection header received
0927 02:51:26.543143 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:26.543150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:26.543204 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:26.543311 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:26.544052 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.545026 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.545801 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.546220 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.546312 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:26.548513 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:26.548517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:26.548518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:26.548530 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:26.550712 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:26.551085 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:26.551088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:26.551088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:26.551179 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:26.551393 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:26.551394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:26.551394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:26.551630 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:26.551774 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:26.551873 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 02:51:26.551998 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260927 02:51:26.561885 17468 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:26.562619 17468 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:26.571300 17468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476: Generated new TSK 0
I20260927 02:51:26.571648 17468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:51:26.606132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39513
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:26.684649 17485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:26.684871 17485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:26.684895 17485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:26.686050 17485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:26.686091 17485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:26.686121 17485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:51:26.687675 17485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:26.687891 17485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:26.688097 17485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:26.692853 17493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:26.692848 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:26.692835 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:26.693112 17485 server_base.cc:1034] running on GCE node
I20260927 02:51:26.693298 17485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:26.693554 17485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:26.694705 17485 hybrid_clock.cc:648] HybridClock initialized: now 1790477486694687 us; error 28 us; skew 500 ppm
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:26.697052 17485 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:51:26.697443 17485 webserver.cc:460] Webserver started at http://127.0.98.1:44111/ using document root <none> and password file <none>
I20260927 02:51:26.697656 17485 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:26.697705 17485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:26.697829 17485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:26.698899 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "f26c9e2ee16647358bd1e67f9152e06f"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "8859c3c403bcc135a6f6828ed881ffa1"
server_key_iv: "d22b8a90208fb1272ba9b15d839a9689"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.699215 17485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f26c9e2ee16647358bd1e67f9152e06f"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "8859c3c403bcc135a6f6828ed881ffa1"
server_key_iv: "d22b8a90208fb1272ba9b15d839a9689"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.700475 17485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:26.701164 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.701301 17485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:26.701381 17485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "f26c9e2ee16647358bd1e67f9152e06f"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "8859c3c403bcc135a6f6828ed881ffa1"
server_key_iv: "d22b8a90208fb1272ba9b15d839a9689"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.701440 17485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:26.719709 17485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:26.721498 17485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:26.721644 17485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:26.721865 17485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:26.722149 17485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:26.722182 17485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.722210 17485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:26.722234 17485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.727685 17485 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41363
I20260927 02:51:26.727742 17613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41363 every 8 connection(s)
I20260927 02:51:26.728021 17485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:51:26.730933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 17485
I20260927 02:51:26.731035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:51:26.731254 392 external_mini_cluster.cc:1428] Setting key a273e9ee2996eb1f8cdca8a4f2abd58b
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:26.737430 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:26.728644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44987 (local address 127.0.98.62:39513)
0927 02:51:26.728705 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:26.728708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:26.728903 (+ 195us) server_negotiation.cc:407] Connection header received
0927 02:51:26.729802 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:26.729803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:26.729814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:26.729874 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:26.730426 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.730988 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.732325 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.732526 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.732577 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:26.734797 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:26.734798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:26.734798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:26.734830 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:26.736227 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:26.736578 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:26.736579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:26.736579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:26.736647 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:26.736838 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:26.736838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:26.736839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:26.736972 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:26.737037 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:26.737247 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 02:51:26.737341 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:51:26.737478 17616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:26.728631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39513 (local address 127.0.98.1:44987)
0927 02:51:26.728789 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:26.728824 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:26.729699 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:26.729937 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:26.729940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:26.730181 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:26.730365 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:26.730368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.731076 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.731077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:26.732254 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:26.732257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.732594 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.732595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:26.732741 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:26.733269 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:26.733277 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:26.734705 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:26.736302 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:26.736308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:26.736309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:26.736511 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:26.736702 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:26.736702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:26.736702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:26.736788 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:26.737025 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:26.737027 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:26.737134 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 02:51:26.737207 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:51:26.737321 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260927 02:51:26.737922 17614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39513
I20260927 02:51:26.738013 17614 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:26.738193 17614 heartbeater.cc:511] Master 127.0.98.62:39513 requested a full tablet report, sending...
I20260927 02:51:26.738772 17423 ts_manager.cc:194] Registered new tserver with Master: f26c9e2ee16647358bd1e67f9152e06f (127.0.98.1:41363)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:51:26.739638 17423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:51:26.753078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39513
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:26.830497 17619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:26.830704 17619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:26.830725 17619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:26.831897 17619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:26.831939 17619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:26.831967 17619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:51:26.833420 17619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:26.833642 17619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:26.833844 17619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:26.838534 17624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:26.838721 17625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:26.838743 17619 server_base.cc:1034] running on GCE node
W20260927 02:51:26.838565 17627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:26.839038 17619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:26.839280 17619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:26.840420 17619 hybrid_clock.cc:648] HybridClock initialized: now 1790477486840408 us; error 29 us; skew 500 ppm
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:26.842901 17619 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:51:26.843310 17619 webserver.cc:460] Webserver started at http://127.0.98.2:37635/ using document root <none> and password file <none>
I20260927 02:51:26.843504 17619 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:26.843545 17619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:26.843660 17619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:26.844655 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "3c954d33d77ea727dc4ff5d37da7c4cb"
server_key_iv: "f71fd046331de0294f3ccf27b1a361b1"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.844967 17619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "3c954d33d77ea727dc4ff5d37da7c4cb"
server_key_iv: "f71fd046331de0294f3ccf27b1a361b1"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.846114 17619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:51:26.846786 17634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.846943 17619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.847023 17619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "3c954d33d77ea727dc4ff5d37da7c4cb"
server_key_iv: "f71fd046331de0294f3ccf27b1a361b1"
server_key_version: "encryptionkey@0"
I20260927 02:51:26.847086 17619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:26.869262 17619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:26.871580 17619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:26.871721 17619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:26.871942 17619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:26.872246 17619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:26.872282 17619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.872309 17619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:26.872344 17619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:26.878034 17619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45505
I20260927 02:51:26.878083 17747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45505 every 8 connection(s)
I20260927 02:51:26.878394 17619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:26.887269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 17619
I20260927 02:51:26.887370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:51:26.887428 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:26.879151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36559 (local address 127.0.98.62:39513)
0927 02:51:26.879231 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:26.879233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:26.879424 (+ 191us) server_negotiation.cc:407] Connection header received
0927 02:51:26.880494 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:26.880495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:26.880507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:26.880555 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:26.881205 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.881691 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.882722 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.882953 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.883006 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:26.884991 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:26.884993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:26.884993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:26.885001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:26.886202 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:26.886543 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:26.886545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:26.886545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:26.886597 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:26.886829 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:26.886829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:26.886830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:26.886939 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:26.886994 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:26.887260 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 02:51:26.887339 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:51:26.887609 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:26.879101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39513 (local address 127.0.98.2:36559)
0927 02:51:26.879269 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:26.879316 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:26.880365 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:26.880635 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:26.880639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:26.880908 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:26.881095 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:26.881100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.881774 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.881775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:26.882629 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:26.882631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:26.883029 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:26.883030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:26.883146 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:26.883581 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:26.883589 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:26.884907 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:26.886271 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:26.886277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:26.886280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:26.886481 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:26.886648 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:26.886649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:26.886649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:26.886752 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:26.886992 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:26.886999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:26.887145 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 02:51:26.887234 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 02:51:26.887385 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260927 02:51:26.887609 392 external_mini_cluster.cc:1428] Setting key 16bf6719fd548d0df665dff9578deee1
I20260927 02:51:26.888114 17748 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39513
I20260927 02:51:26.888240 17748 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:26.888415 17748 heartbeater.cc:511] Master 127.0.98.62:39513 requested a full tablet report, sending...
I20260927 02:51:26.888761 17423 ts_manager.cc:194] Registered new tserver with Master: 4c4d3e8cdf3f44219cb220af9dd4fd89 (127.0.98.2:45505)
I20260927 02:51:26.889201 17423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36559
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:51:26.909926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39513
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:26.988674 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:26.988885 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:26.988917 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:26.990052 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:26.990092 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:26.990127 17753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:51:26.991663 17753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:26.991904 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:26.992120 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:26.996680 17759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:26.996688 17758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:26.996688 17761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:26.997046 17753 server_base.cc:1034] running on GCE node
I20260927 02:51:26.997223 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:26.997422 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:26.998572 17753 hybrid_clock.cc:648] HybridClock initialized: now 1790477486998552 us; error 29 us; skew 500 ppm
Sep 27 02:51:26 dist-test-slave-f4rg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:27.000869 17753 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:51:27.001314 17753 webserver.cc:460] Webserver started at http://127.0.98.3:36793/ using document root <none> and password file <none>
I20260927 02:51:27.001516 17753 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:27.001565 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:27.001689 17753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:27.002977 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "6aa7012879ea408c91d2c8d399cce448"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "bb15e917948660d4be2c39c292813a53"
server_key_iv: "7790a4372458179d088f630951384cc5"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.003379 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6aa7012879ea408c91d2c8d399cce448"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "bb15e917948660d4be2c39c292813a53"
server_key_iv: "7790a4372458179d088f630951384cc5"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.004581 17753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:51:27.005189 17768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.005368 17753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.005440 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "6aa7012879ea408c91d2c8d399cce448"
format_stamp: "Formatted at 2026-09-27 02:51:26 on dist-test-slave-f4rg"
server_key: "bb15e917948660d4be2c39c292813a53"
server_key_iv: "7790a4372458179d088f630951384cc5"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.005491 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:27.013870 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:27.015645 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:27.015771 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:27.015966 17753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:27.016254 17753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:27.016292 17753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.016319 17753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:27.016343 17753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.021833 17753 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36971
I20260927 02:51:27.021871 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36971 every 8 connection(s)
I20260927 02:51:27.022238 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:51:27.024219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 17753
I20260927 02:51:27.024317 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:51:27.024459 392 external_mini_cluster.cc:1428] Setting key 913fc33dbeac4afe940613e8b8ab1079
Sep 27 02:51:27 dist-test-slave-f4rg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:27.031647 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.022916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45439 (local address 127.0.98.62:39513)
0927 02:51:27.022973 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.022975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.023178 (+ 203us) server_negotiation.cc:407] Connection header received
0927 02:51:27.024596 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.024598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.024610 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.024682 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:27.025454 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.025975 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.027012 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.027198 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.027249 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.029259 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:27.029260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:27.029260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:27.029266 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:27.030507 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:27.030888 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:27.030889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:27.030890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:27.030937 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:27.031138 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:27.031138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:27.031138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:27.031236 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:27.031279 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.031518 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.031584 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:51:27.031834 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.022891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39513 (local address 127.0.98.3:45439)
0927 02:51:27.023061 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.023102 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.023997 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.024790 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.024796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.025151 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:27.025388 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.025392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.026049 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.026049 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.026954 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.026957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.027289 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.027290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.027405 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.027868 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:27.027876 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:27.029177 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:27.030588 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:27.030594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:27.030595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:27.030798 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:27.030988 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:27.030989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:27.030989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:27.031089 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:27.031285 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:27.031289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:27.031409 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.031491 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.031626 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260927 02:51:27.032274 17882 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39513
I20260927 02:51:27.032358 17882 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:27.032516 17882 heartbeater.cc:511] Master 127.0.98.62:39513 requested a full tablet report, sending...
I20260927 02:51:27.032828 17423 ts_manager.cc:194] Registered new tserver with Master: 6aa7012879ea408c91d2c8d399cce448 (127.0.98.3:36971)
I20260927 02:51:27.033190 17423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45439
I20260927 02:51:27.035944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:51:27.042172 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.037146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.0.98.62:39513)
0927 02:51:27.037204 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.037206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.037298 (+ 92us) server_negotiation.cc:407] Connection header received
0927 02:51:27.037341 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.037343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.037352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.037410 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:27.037892 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.038359 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.039060 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.039242 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.039297 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.040343 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:27.040344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:27.040344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:27.040350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:27.041367 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:27.041625 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:27.041626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:27.041627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:27.041668 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:27.041792 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:27.041793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:27.041793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:27.041907 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:27.041971 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.042042 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.042101 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:51:27.043622 17423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:51:27.045048 17423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:51:27.051007 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.047155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.98.2:45505)
0927 02:51:27.047314 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.047319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.047330 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:51:27.047370 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.047372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.047388 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.047481 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:27.047628 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.048406 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.049827 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.050581 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.050640 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.050701 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.050886 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.050925 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":67,"threads_started":1}
I20260927 02:51:27.050999 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.046903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45505 (local address 127.0.0.1:38242)
0927 02:51:27.047054 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.047099 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.047162 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.047490 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.047493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.047496 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:27.047562 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.047564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.048491 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.048492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.049759 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.049761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.050681 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.050683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.050767 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.050781 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.050867 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.050925 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260927 02:51:27.051187 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.046903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36971 (local address 127.0.0.1:43960)
0927 02:51:27.047311 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.047321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.047369 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.047553 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.047554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.047556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:27.047610 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.047612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.048971 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.048972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.050087 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.050089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.050915 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.050916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.051023 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.051027 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.051055 (+ 28us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.051095 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":20,"thread_start_us":49,"threads_started":1}
I20260927 02:51:27.051507 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.047039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.98.3:36971)
0927 02:51:27.047371 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.047375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.047402 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:51:27.047431 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.047432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.047462 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.047681 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:27.047757 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.048839 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.050153 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.050828 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.051295 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.051364 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.051389 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.051431 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":89,"threads_started":1}
I20260927 02:51:27.051852 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.046913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41363 (local address 127.0.0.1:43212)
0927 02:51:27.047426 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.047434 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.047490 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.047820 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.047821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.047823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:27.047861 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.047862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.049156 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.049157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.050319 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.050322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.051406 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.051407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.051686 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.051691 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.051747 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.051790 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":51,"threads_started":1}
I20260927 02:51:27.052182 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.047167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.0.98.1:41363)
0927 02:51:27.047316 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.047320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.047542 (+ 222us) server_negotiation.cc:407] Connection header received
0927 02:51:27.047584 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.047586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.047663 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.047734 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:27.047922 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.049066 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.050399 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.051342 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.051405 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.051452 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.052045 (+ 593us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.052102 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260927 02:51:27.052117 17682 tablet_service.cc:1467] Processing CreateTablet for tablet b6bcf5cd236c4d20b872b840be00e684 (DEFAULT_TABLE table=test-table [id=b25bc3d73912463eb90a9c348b135b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:27.052114 17816 tablet_service.cc:1467] Processing CreateTablet for tablet b6bcf5cd236c4d20b872b840be00e684 (DEFAULT_TABLE table=test-table [id=b25bc3d73912463eb90a9c348b135b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:27.052448 17816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bcf5cd236c4d20b872b840be00e684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:27.052448 17682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bcf5cd236c4d20b872b840be00e684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:27.052748 17548 tablet_service.cc:1467] Processing CreateTablet for tablet b6bcf5cd236c4d20b872b840be00e684 (DEFAULT_TABLE table=test-table [id=b25bc3d73912463eb90a9c348b135b25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:27.052984 17548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6bcf5cd236c4d20b872b840be00e684. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:27.055476 17896 tablet_bootstrap.cc:492] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: Bootstrap starting.
I20260927 02:51:27.055871 17897 tablet_bootstrap.cc:492] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Bootstrap starting.
I20260927 02:51:27.055956 17898 tablet_bootstrap.cc:492] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Bootstrap starting.
I20260927 02:51:27.056094 17896 tablet_bootstrap.cc:654] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:27.056531 17896 log.cc:824] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:27.056540 17897 tablet_bootstrap.cc:654] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:27.056540 17898 tablet_bootstrap.cc:654] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:27.057072 17898 log.cc:824] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:27.057072 17897 log.cc:824] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:27.057144 17896 tablet_bootstrap.cc:492] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: No bootstrap required, opened a new log
I20260927 02:51:27.057225 17896 ts_tablet_manager.cc:1397] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:51:27.057742 17898 tablet_bootstrap.cc:492] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: No bootstrap required, opened a new log
I20260927 02:51:27.057767 17897 tablet_bootstrap.cc:492] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: No bootstrap required, opened a new log
I20260927 02:51:27.057812 17897 ts_tablet_manager.cc:1397] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:27.057812 17898 ts_tablet_manager.cc:1397] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:27.058887 17896 raft_consensus.cc:348] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.059029 17896 raft_consensus.cc:374] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:27.059062 17896 raft_consensus.cc:729] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26c9e2ee16647358bd1e67f9152e06f, State: Initialized, Role: FOLLOWER
I20260927 02:51:27.059195 17896 consensus_queue.cc:260] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.059461 17896 ts_tablet_manager.cc:1428] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:51:27.059582 17614 heartbeater.cc:503] Master 127.0.98.62:39513 was elected leader, sending a full tablet report...
I20260927 02:51:27.059680 17897 raft_consensus.cc:348] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.059696 17898 raft_consensus.cc:348] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.059847 17898 raft_consensus.cc:374] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:27.059890 17898 raft_consensus.cc:729] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4d3e8cdf3f44219cb220af9dd4fd89, State: Initialized, Role: FOLLOWER
I20260927 02:51:27.059887 17897 raft_consensus.cc:374] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:27.059922 17897 raft_consensus.cc:729] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa7012879ea408c91d2c8d399cce448, State: Initialized, Role: FOLLOWER
I20260927 02:51:27.059991 17898 consensus_queue.cc:260] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.060027 17897 consensus_queue.cc:260] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.060252 17897 ts_tablet_manager.cc:1428] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:51:27.060252 17898 ts_tablet_manager.cc:1428] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:51:27.060261 17748 heartbeater.cc:503] Master 127.0.98.62:39513 was elected leader, sending a full tablet report...
I20260927 02:51:27.060353 17882 heartbeater.cc:503] Master 127.0.98.62:39513 was elected leader, sending a full tablet report...
W20260927 02:51:27.128932 17749 tablet.cc:2369] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:51:27.228737 17615 tablet.cc:2369] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:51:27.272759 17883 tablet.cc:2369] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:51:27.351112 17902 raft_consensus.cc:482] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:51:27.351238 17902 raft_consensus.cc:504] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.351589 17902 leader_election.cc:288] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aa7012879ea408c91d2c8d399cce448 (127.0.98.3:36971), 4c4d3e8cdf3f44219cb220af9dd4fd89 (127.0.98.2:45505)
I20260927 02:51:27.354250 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.351817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58653 (local address 127.0.98.2:45505)
0927 02:51:27.351861 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.351863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.351951 (+ 88us) server_negotiation.cc:407] Connection header received
0927 02:51:27.352004 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.352005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.352017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.352064 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:27.352202 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.352662 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.353582 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.353947 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.354004 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.354015 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.354152 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.354182 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:51:27.354297 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.351759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45505 (local address 127.0.98.1:58653)
0927 02:51:27.351878 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.351888 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.351944 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.352068 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.352069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.352071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:27.352143 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.352149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.352735 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.352736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.353504 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.353506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.354023 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.354024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.354094 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.354098 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.354142 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.354182 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
I20260927 02:51:27.354302 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.351850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40691 (local address 127.0.98.3:36971)
0927 02:51:27.351893 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.351895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.351946 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:51:27.352015 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.352017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.352025 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.352071 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:27.352228 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.352672 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.353656 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.354007 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.354062 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.354071 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.354204 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.354238 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:51:27.354377 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.351762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36971 (local address 127.0.98.1:40691)
0927 02:51:27.351890 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.351900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.351959 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.352082 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.352082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.352084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:27.352165 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.352166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.352742 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.352743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.353570 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.353573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.354076 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.354076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.354143 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.354147 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.354190 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.354227 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":34,"thread_start_us":60,"threads_started":1}
I20260927 02:51:27.354728 17836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bcf5cd236c4d20b872b840be00e684" candidate_uuid: "f26c9e2ee16647358bd1e67f9152e06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa7012879ea408c91d2c8d399cce448" is_pre_election: true
I20260927 02:51:27.354897 17836 raft_consensus.cc:2454] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26c9e2ee16647358bd1e67f9152e06f in term 0.
I20260927 02:51:27.355115 17503 leader_election.cc:302] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aa7012879ea408c91d2c8d399cce448, f26c9e2ee16647358bd1e67f9152e06f; no voters:
I20260927 02:51:27.355123 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bcf5cd236c4d20b872b840be00e684" candidate_uuid: "f26c9e2ee16647358bd1e67f9152e06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" is_pre_election: true
I20260927 02:51:27.355254 17702 raft_consensus.cc:2454] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f26c9e2ee16647358bd1e67f9152e06f in term 0.
I20260927 02:51:27.355250 17902 raft_consensus.cc:2781] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:51:27.355293 17902 raft_consensus.cc:482] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:51:27.355314 17902 raft_consensus.cc:3037] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:27.356003 17902 raft_consensus.cc:504] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.356181 17902 leader_election.cc:288] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [CANDIDATE]: Term 1 election: Requested vote from peers 6aa7012879ea408c91d2c8d399cce448 (127.0.98.3:36971), 4c4d3e8cdf3f44219cb220af9dd4fd89 (127.0.98.2:45505)
I20260927 02:51:27.356276 17702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bcf5cd236c4d20b872b840be00e684" candidate_uuid: "f26c9e2ee16647358bd1e67f9152e06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89"
I20260927 02:51:27.356352 17702 raft_consensus.cc:3037] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:27.357226 17702 raft_consensus.cc:2454] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26c9e2ee16647358bd1e67f9152e06f in term 1.
I20260927 02:51:27.357402 17501 leader_election.cc:302] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c4d3e8cdf3f44219cb220af9dd4fd89, f26c9e2ee16647358bd1e67f9152e06f; no voters:
I20260927 02:51:27.357509 17902 raft_consensus.cc:2781] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:27.357658 17902 raft_consensus.cc:686] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [term 1 LEADER]: Becoming Leader. State: Replica: f26c9e2ee16647358bd1e67f9152e06f, State: Running, Role: LEADER
I20260927 02:51:27.357786 17902 consensus_queue.cc:237] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } }
I20260927 02:51:27.359959 17836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6bcf5cd236c4d20b872b840be00e684" candidate_uuid: "f26c9e2ee16647358bd1e67f9152e06f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa7012879ea408c91d2c8d399cce448"
I20260927 02:51:27.360045 17836 raft_consensus.cc:3037] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:27.360837 17836 raft_consensus.cc:2454] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f26c9e2ee16647358bd1e67f9152e06f in term 1.
I20260927 02:51:27.360841 17422 catalog_manager.cc:5495] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f reported cstate change: term changed from 0 to 1, leader changed from <none> to f26c9e2ee16647358bd1e67f9152e06f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f26c9e2ee16647358bd1e67f9152e06f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f26c9e2ee16647358bd1e67f9152e06f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41363 } health_report { overall_health: HEALTHY } } }
I20260927 02:51:27.402112 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.399601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.0.98.1:41363)
0927 02:51:27.399656 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.399658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.399680 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:51:27.399716 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.399717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.399737 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.399790 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:27.399941 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.400411 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.401207 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.401498 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.401573 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.401757 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:27.401900 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:27.401999 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.402026 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.402048 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:51:27.403990 17836 raft_consensus.cc:1261] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 1 FOLLOWER]: Refusing update from remote peer f26c9e2ee16647358bd1e67f9152e06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:27.404036 17702 raft_consensus.cc:1261] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 1 FOLLOWER]: Refusing update from remote peer f26c9e2ee16647358bd1e67f9152e06f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:27.404320 17902 consensus_queue.cc:1035] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c4d3e8cdf3f44219cb220af9dd4fd89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:27.404465 17907 consensus_queue.cc:1035] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6aa7012879ea408c91d2c8d399cce448" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:27.407095 17912 mvcc.cc:205] Tried to move back new op lower bound from 7333795788404011008 to 7333795788218118144. Current Snapshot: MvccSnapshot[applied={T|T < 7333795788404011008}]
I20260927 02:51:27.407382 17911 mvcc.cc:205] Tried to move back new op lower bound from 7333795788404011008 to 7333795788218118144. Current Snapshot: MvccSnapshot[applied={T|T < 7333795788404011008}]
I20260927 02:51:27.408740 17422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:27.408865 17422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:27.409624 17422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28983ff0217640b7bad6d7b533c51476: Sending DeleteTablet for 3 replicas of tablet b6bcf5cd236c4d20b872b840be00e684
I20260927 02:51:27.409927 17548 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bcf5cd236c4d20b872b840be00e684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43212
I20260927 02:51:27.409940 17816 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bcf5cd236c4d20b872b840be00e684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43960
I20260927 02:51:27.410169 17920 tablet_replica.cc:330] T b6bcf5cd236c4d20b872b840be00e684 P f26c9e2ee16647358bd1e67f9152e06f: stopping tablet replica
I20260927 02:51:27.410161 17682 tablet_service.cc:1514] Processing DeleteTablet for tablet b6bcf5cd236c4d20b872b840be00e684 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38242
I20260927 02:51:27.410223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 17485
I20260927 02:51:27.410244 17921 tablet_replica.cc:330] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: stopping tablet replica
I20260927 02:51:27.410332 17921 raft_consensus.cc:2229] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:27.410420 17921 raft_consensus.cc:2258] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:27.411221 17921 ts_tablet_manager.cc:1905] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:27.411386 17922 tablet_replica.cc:330] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: stopping tablet replica
I20260927 02:51:27.411474 17922 raft_consensus.cc:2229] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:27.411571 17922 raft_consensus.cc:2258] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:27.411909 17922 ts_tablet_manager.cc:1905] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:27.413205 17922 ts_tablet_manager.cc:1918] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:27.413252 17922 log.cc:1196] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/b6bcf5cd236c4d20b872b840be00e684
I20260927 02:51:27.413232 17921 ts_tablet_manager.cc:1918] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:27.413292 17921 log.cc:1196] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/b6bcf5cd236c4d20b872b840be00e684
I20260927 02:51:27.413420 17922 ts_tablet_manager.cc:1939] T b6bcf5cd236c4d20b872b840be00e684 P 4c4d3e8cdf3f44219cb220af9dd4fd89: Deleting consensus metadata
I20260927 02:51:27.413525 17921 ts_tablet_manager.cc:1939] T b6bcf5cd236c4d20b872b840be00e684 P 6aa7012879ea408c91d2c8d399cce448: Deleting consensus metadata
I20260927 02:51:27.413841 17407 catalog_manager.cc:4841] TS 4c4d3e8cdf3f44219cb220af9dd4fd89 (127.0.98.2:45505): tablet b6bcf5cd236c4d20b872b840be00e684 (table test-table [id=b25bc3d73912463eb90a9c348b135b25]) successfully deleted
I20260927 02:51:27.413869 17409 catalog_manager.cc:4841] TS 6aa7012879ea408c91d2c8d399cce448 (127.0.98.3:36971): tablet b6bcf5cd236c4d20b872b840be00e684 (table test-table [id=b25bc3d73912463eb90a9c348b135b25]) successfully deleted
W20260927 02:51:27.415566 17408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41363 (error 108)
I20260927 02:51:27.415716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 17619
I20260927 02:51:27.416965 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.416840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41363 (local address 127.0.0.1:43232)
0927 02:51:27.416900 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.416936 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 02:51:27.417098 17408 catalog_manager.cc:4584] TS f26c9e2ee16647358bd1e67f9152e06f (127.0.98.1:41363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6bcf5cd236c4d20b872b840be00e684: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41363: connect: Connection refused (error 111)
I20260927 02:51:27.420100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 17753
I20260927 02:51:27.424644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 17391
2026-09-27T02:51:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2052 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T02:51:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:27Z Disabled control of system clock
I20260927 02:51:27.450275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38219
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:51:27.529089 17927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:27.529316 17927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:27.529336 17927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:27.530695 17927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:27.530754 17927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:27.530771 17927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:27.530781 17927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:27.530792 17927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:27.532276 17927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:27.532531 17927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:27.532812 17927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:27.537497 17933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:27.537568 17932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:27.537497 17935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:27.537632 17927 server_base.cc:1034] running on GCE node
I20260927 02:51:27.537946 17927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:27.538209 17927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:27.539342 17927 hybrid_clock.cc:648] HybridClock initialized: now 1790477487539325 us; error 34 us; skew 500 ppm
I20260927 02:51:27.540504 17927 webserver.cc:460] Webserver started at http://127.0.98.62:34487/ using document root <none> and password file <none>
I20260927 02:51:27.540719 17927 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:27.540766 17927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:27.540892 17927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:27.541921 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "7ff438f75c49490aa5442f66386b40a8"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "fd122deb0a34cfa99b153a3cfb24437b"
server_key_iv: "a2931f55320c558783281153c3d25248"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.542245 17927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ff438f75c49490aa5442f66386b40a8"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "fd122deb0a34cfa99b153a3cfb24437b"
server_key_iv: "a2931f55320c558783281153c3d25248"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.543476 17927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:51:27.544130 17941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.544276 17927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:27.544353 17927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "7ff438f75c49490aa5442f66386b40a8"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "fd122deb0a34cfa99b153a3cfb24437b"
server_key_iv: "a2931f55320c558783281153c3d25248"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.544414 17927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:27.562476 17927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:27.562894 17927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:27.563032 17927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:27.566676 17927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38219
I20260927 02:51:27.566731 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38219 every 8 connection(s)
I20260927 02:51:27.567050 17927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:27.567999 17994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:27.570163 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8: Bootstrap starting.
I20260927 02:51:27.570677 17994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:27.570935 17994 log.cc:824] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:27.571504 17994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8: No bootstrap required, opened a new log
I20260927 02:51:27.573022 17994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff438f75c49490aa5442f66386b40a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38219 } }
I20260927 02:51:27.573156 17994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:27.573176 17994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff438f75c49490aa5442f66386b40a8, State: Initialized, Role: FOLLOWER
I20260927 02:51:27.573284 17994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff438f75c49490aa5442f66386b40a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38219 } }
I20260927 02:51:27.573355 17994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:27.573390 17994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:27.573424 17994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:27.573880 17994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff438f75c49490aa5442f66386b40a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38219 } }
I20260927 02:51:27.573966 17994 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff438f75c49490aa5442f66386b40a8; no voters:
I20260927 02:51:27.574087 17994 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:27.574146 17997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:27.574299 17994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:27.574329 17997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff438f75c49490aa5442f66386b40a8, State: Running, Role: LEADER
I20260927 02:51:27.574499 17997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff438f75c49490aa5442f66386b40a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38219 } }
I20260927 02:51:27.574538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 17927
I20260927 02:51:27.574626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:27.574842 392 external_mini_cluster.cc:1428] Setting key d73807c1201ee583b13f1016d10e6951
I20260927 02:51:27.575250 17999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff438f75c49490aa5442f66386b40a8. Latest consensus state: current_term: 1 leader_uuid: "7ff438f75c49490aa5442f66386b40a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff438f75c49490aa5442f66386b40a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38219 } } }
I20260927 02:51:27.575342 17999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:27.575591 17998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff438f75c49490aa5442f66386b40a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff438f75c49490aa5442f66386b40a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38219 } } }
I20260927 02:51:27.575657 17998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:27.576023 18007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:27.576634 18007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:27.577951 18007 catalog_manager.cc:1347] Generated new cluster ID: 94672c3df05d4752ba859b8268e11ae0
I20260927 02:51:27.578011 18007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:27.581069 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.575484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.0.98.62:38219)
0927 02:51:27.575876 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.575886 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.575921 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:51:27.576050 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.576058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.576130 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.576275 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:27.576560 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.577914 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.579036 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.579645 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.579756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.580399 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:27.580402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:27.580403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:27.580621 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:27.580660 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:27.580674 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:27.580731 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.580798 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.580870 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":91,"threads_started":1}
I20260927 02:51:27.600436 18007 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:27.601168 18007 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:27.608469 18007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8: Generated new TSK 0
I20260927 02:51:27.608673 18007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:27.610348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38219
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:51:27.688723 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:27.688926 18018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:27.688954 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:27.690056 18018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:27.690094 18018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:27.690126 18018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:51:27.691473 18018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:27.691704 18018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:27.691964 18018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:27.696506 18024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:27.696506 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:27.696519 18026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:27.696883 18018 server_base.cc:1034] running on GCE node
I20260927 02:51:27.697057 18018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:27.697271 18018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:27.698417 18018 hybrid_clock.cc:648] HybridClock initialized: now 1790477487698404 us; error 31 us; skew 500 ppm
I20260927 02:51:27.699680 18018 webserver.cc:460] Webserver started at http://127.0.98.1:39461/ using document root <none> and password file <none>
I20260927 02:51:27.699896 18018 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:27.699940 18018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:27.700057 18018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:27.701099 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "6a56a978666e4e611b6a990eb5444688"
server_key_iv: "3e223e04af7e5e455676c9257c3c8ba8"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.701418 18018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "6a56a978666e4e611b6a990eb5444688"
server_key_iv: "3e223e04af7e5e455676c9257c3c8ba8"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.702729 18018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:27.703576 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.703778 18018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:27.703866 18018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "6a56a978666e4e611b6a990eb5444688"
server_key_iv: "3e223e04af7e5e455676c9257c3c8ba8"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.703925 18018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:27.718861 18018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:27.719162 18018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:27.719266 18018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:27.719463 18018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:27.719722 18018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:27.719749 18018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.719770 18018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:27.719785 18018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.725001 18018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43887
I20260927 02:51:27.725080 18145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43887 every 8 connection(s)
I20260927 02:51:27.725373 18018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:51:27.730337 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.726057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41699 (local address 127.0.98.62:38219)
0927 02:51:27.726144 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.726146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.726343 (+ 197us) server_negotiation.cc:407] Connection header received
0927 02:51:27.726837 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.726838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.726851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.726907 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:27.727592 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.728081 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.729093 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.729286 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.729334 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.730132 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:27.730134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:27.730134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:27.730139 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:27.730154 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:27.730158 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:27.730194 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.730253 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.730279 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 02:51:27.730575 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.726032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38219 (local address 127.0.98.1:41699)
0927 02:51:27.726222 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.726257 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.726744 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.726974 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.726977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.726987 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:27.727517 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.727522 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.728145 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.728146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.729024 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.729027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.729363 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.729364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.729482 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.729990 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:27.729990 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:27.730022 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:27.730045 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:27.730200 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:27.730203 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:27.730205 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.730252 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.730327 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260927 02:51:27.731042 18146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38219
I20260927 02:51:27.731140 18146 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:27.731310 18146 heartbeater.cc:511] Master 127.0.98.62:38219 requested a full tablet report, sending...
I20260927 02:51:27.731855 17958 ts_manager.cc:194] Registered new tserver with Master: 48b69f465e9b40f9b1ed4b7c276dc0d4 (127.0.98.1:43887)
I20260927 02:51:27.732425 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41699
I20260927 02:51:27.735071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18018
I20260927 02:51:27.735162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:51:27.735308 392 external_mini_cluster.cc:1428] Setting key 407c83524c44644b3140b3249f6e6ca2
I20260927 02:51:27.736256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38219
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:51:27.813732 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:27.813935 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:27.813964 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:27.815122 18149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:27.815162 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:27.815194 18149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:51:27.816527 18149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:27.816767 18149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:27.817044 18149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:27.821727 18155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:27.821717 18154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:27.821717 18157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:27.821868 18149 server_base.cc:1034] running on GCE node
I20260927 02:51:27.822109 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:27.822329 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:27.823454 18149 hybrid_clock.cc:648] HybridClock initialized: now 1790477487823440 us; error 32 us; skew 500 ppm
I20260927 02:51:27.824611 18149 webserver.cc:460] Webserver started at http://127.0.98.2:46601/ using document root <none> and password file <none>
I20260927 02:51:27.824841 18149 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:27.824887 18149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:27.824991 18149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:27.826020 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "dd60c3bb32a3460dabbf80eeaaa11332"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "3e9353fc218fba317854ca102495d0c6"
server_key_iv: "99bbb5f736d1d62daea59ee62407196f"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.826340 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dd60c3bb32a3460dabbf80eeaaa11332"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "3e9353fc218fba317854ca102495d0c6"
server_key_iv: "99bbb5f736d1d62daea59ee62407196f"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.827699 18149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:51:27.828424 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.828615 18149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:27.828692 18149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "dd60c3bb32a3460dabbf80eeaaa11332"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "3e9353fc218fba317854ca102495d0c6"
server_key_iv: "99bbb5f736d1d62daea59ee62407196f"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.828759 18149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:27.848963 18149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:27.849261 18149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:27.849386 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:27.849591 18149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:27.849906 18149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:27.849941 18149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.849969 18149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:27.849987 18149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.855266 18149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46485
I20260927 02:51:27.855319 18276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46485 every 8 connection(s)
I20260927 02:51:27.855614 18149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:51:27.860375 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.856282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49821 (local address 127.0.98.62:38219)
0927 02:51:27.856339 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.856340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.856538 (+ 198us) server_negotiation.cc:407] Connection header received
0927 02:51:27.857033 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.857039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.857050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.857108 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:27.857722 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.858220 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.859187 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.859379 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.859424 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.860164 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:27.860165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:27.860166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:27.860169 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:27.860185 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:27.860190 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:27.860231 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:27.860290 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:51:27.860317 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:51:27.860558 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.856281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38219 (local address 127.0.98.2:49821)
0927 02:51:27.856434 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.856466 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.856921 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.857161 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.857165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.857172 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:27.857656 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.857660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.858279 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.858280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.859112 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:27.859114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:27.859432 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:27.859433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:27.859535 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:27.860031 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:27.860032 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:27.860060 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:27.860091 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:27.860238 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:27.860240 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:27.860242 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:51:27.860288 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:51:27.860363 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260927 02:51:27.860704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18149
I20260927 02:51:27.860808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:51:27.861006 392 external_mini_cluster.cc:1428] Setting key 14b979d60ba5901b527ee03a0ebffaec
I20260927 02:51:27.861027 18277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38219
I20260927 02:51:27.861111 18277 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:27.861301 18277 heartbeater.cc:511] Master 127.0.98.62:38219 requested a full tablet report, sending...
I20260927 02:51:27.861629 17958 ts_manager.cc:194] Registered new tserver with Master: dd60c3bb32a3460dabbf80eeaaa11332 (127.0.98.2:46485)
I20260927 02:51:27.861888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38219
--builtin_ntp_servers=127.0.98.20:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:51:27.862025 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49821
W20260927 02:51:27.945346 18280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:27.945549 18280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:27.945569 18280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:27.946724 18280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:27.946764 18280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:27.946791 18280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:51:27.948230 18280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38219
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:27.948449 18280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:27.948711 18280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:27.953431 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:27.953410 18285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:27.953482 18280 server_base.cc:1034] running on GCE node
W20260927 02:51:27.953419 18286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:27.953811 18280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:27.954041 18280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:27.955230 18280 hybrid_clock.cc:648] HybridClock initialized: now 1790477487955215 us; error 29 us; skew 500 ppm
I20260927 02:51:27.956439 18280 webserver.cc:460] Webserver started at http://127.0.98.3:39229/ using document root <none> and password file <none>
I20260927 02:51:27.956722 18280 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:27.956769 18280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:27.956889 18280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:27.957950 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c2b0cf5cc904e58807412c22ae4002b"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "e66ea421b741ac716b29d82deb7a77fc"
server_key_iv: "826fed56b4638042e6bce9f63d478458"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.958266 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c2b0cf5cc904e58807412c22ae4002b"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "e66ea421b741ac716b29d82deb7a77fc"
server_key_iv: "826fed56b4638042e6bce9f63d478458"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.959623 18280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:27.960263 18294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.960391 18280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:27.960467 18280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "9c2b0cf5cc904e58807412c22ae4002b"
format_stamp: "Formatted at 2026-09-27 02:51:27 on dist-test-slave-f4rg"
server_key: "e66ea421b741ac716b29d82deb7a77fc"
server_key_iv: "826fed56b4638042e6bce9f63d478458"
server_key_version: "encryptionkey@0"
I20260927 02:51:27.960531 18280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:27.991400 18280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:27.991778 18280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:27.991912 18280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:27.992146 18280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:27.992462 18280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:27.992503 18280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.992534 18280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:27.992555 18280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:27.997764 18280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42211
I20260927 02:51:27.997819 18407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42211 every 8 connection(s)
I20260927 02:51:27.998128 18280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:51:27.999876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18280
I20260927 02:51:27.999971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:51:28.000155 392 external_mini_cluster.cc:1428] Setting key cc448e0b9d6b865b4103f207c1505dd6
I20260927 02:51:28.003095 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.998794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35259 (local address 127.0.98.62:38219)
0927 02:51:27.998882 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:27.998884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:27.999082 (+ 198us) server_negotiation.cc:407] Connection header received
0927 02:51:27.999630 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:27.999631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:27.999641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:27.999687 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:28.000404 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.000955 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.001908 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.002100 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.002153 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.002869 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:28.002870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:28.002871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:28.002875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:28.002893 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:28.002897 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:28.002949 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.003013 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.003038 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 02:51:28.003259 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:27.998767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38219 (local address 127.0.98.3:35259)
0927 02:51:27.998963 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:27.999000 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:27.999469 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:27.999750 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:27.999753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:27.999762 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:28.000307 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.000314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.001022 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.001023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.001825 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.001827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.002159 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.002160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.002262 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.002709 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:28.002710 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:28.002738 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:28.002760 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:28.002952 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:28.002954 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:28.002956 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:51:28.003003 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:51:28.003078 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260927 02:51:28.003708 18408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38219
I20260927 02:51:28.003798 18408 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:28.003961 18408 heartbeater.cc:511] Master 127.0.98.62:38219 requested a full tablet report, sending...
I20260927 02:51:28.004295 17958 ts_manager.cc:194] Registered new tserver with Master: 9c2b0cf5cc904e58807412c22ae4002b (127.0.98.3:42211)
I20260927 02:51:28.004618 17958 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35259
I20260927 02:51:28.011893 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:51:28.016008 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.013224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.0.98.62:38219)
0927 02:51:28.013292 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:28.013294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:28.013372 (+ 78us) server_negotiation.cc:407] Connection header received
0927 02:51:28.013427 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:28.013428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:28.013441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:28.013491 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:28.013690 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.014170 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.014916 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.015114 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.015169 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.015761 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:28.015762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:28.015762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:28.015766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:28.015782 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:28.015786 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:28.015830 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.015891 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.015916 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:51:28.017513 17958 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:51:28.019089 17958 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:51:28.025169 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.021249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.0.98.3:42211)
0927 02:51:28.021622 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:28.021626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:28.021636 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:51:28.021676 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:28.021678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:28.021694 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:28.021787 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:28.021930 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.023072 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.024278 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.024765 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.024826 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.024915 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.025046 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.025085 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":82,"threads_started":1}
I20260927 02:51:28.025168 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.021112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42211 (local address 127.0.0.1:50024)
0927 02:51:28.021244 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:28.021285 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:28.021344 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:28.021772 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:28.021774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:28.021776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:28.021847 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.021849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.023165 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.023166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.024211 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.024212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.024873 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.024874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.024950 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.024962 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:51:28.025026 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 02:51:28.025080 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":26,"thread_start_us":82,"threads_started":1}
I20260927 02:51:28.025346 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.021419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.0.98.1:43887)
0927 02:51:28.021594 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:28.021598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:28.021653 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:51:28.021730 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:28.021731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:28.021743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:28.021809 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:28.022002 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.022761 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.024142 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.024856 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.024923 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.025007 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.025227 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.025269 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260927 02:51:28.025483 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.021121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43887 (local address 127.0.0.1:46424)
0927 02:51:28.021594 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:28.021604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:28.021666 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:28.021882 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:28.021883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:28.021885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:28.021930 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.021932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.022852 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.022853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.024064 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.024066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.025042 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.025043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.025101 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.025105 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:28.025147 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:51:28.025183 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":111,"threads_started":1}
I20260927 02:51:28.026098 18080 tablet_service.cc:1467] Processing CreateTablet for tablet 9a25cd0de0394cf1ac5eb45f1c173944 (DEFAULT_TABLE table=test-table [id=d15cc5e7ef264eeda1ed6e15f9e111b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:28.026110 18342 tablet_service.cc:1467] Processing CreateTablet for tablet 9a25cd0de0394cf1ac5eb45f1c173944 (DEFAULT_TABLE table=test-table [id=d15cc5e7ef264eeda1ed6e15f9e111b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:28.026451 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a25cd0de0394cf1ac5eb45f1c173944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:28.026455 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a25cd0de0394cf1ac5eb45f1c173944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:28.026924 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.021334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46485 (local address 127.0.0.1:59556)
0927 02:51:28.021870 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:28.021879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:28.021923 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:28.022110 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:28.022111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:28.022113 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:28.022144 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.022145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.023336 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.023337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.024225 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.024227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.025162 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.025163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.025214 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.025217 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:28.025535 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 02:51:28.025582 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":4,"thread_start_us":63,"threads_started":1}
I20260927 02:51:28.027235 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.021419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.0.98.2:46485)
0927 02:51:28.021593 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:28.021598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:28.021973 (+ 375us) server_negotiation.cc:407] Connection header received
0927 02:51:28.022026 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:28.022028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:28.022043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:28.022113 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:28.022200 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.023265 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.024312 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.024988 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.025044 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.025133 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.027094 (+ 1961us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.027143 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260927 02:51:28.027871 18211 tablet_service.cc:1467] Processing CreateTablet for tablet 9a25cd0de0394cf1ac5eb45f1c173944 (DEFAULT_TABLE table=test-table [id=d15cc5e7ef264eeda1ed6e15f9e111b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:28.028206 18211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a25cd0de0394cf1ac5eb45f1c173944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:28.029260 18422 tablet_bootstrap.cc:492] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Bootstrap starting.
I20260927 02:51:28.029618 18423 tablet_bootstrap.cc:492] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4: Bootstrap starting.
I20260927 02:51:28.029995 18422 tablet_bootstrap.cc:654] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:28.030215 18423 tablet_bootstrap.cc:654] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:28.030296 18422 log.cc:824] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:28.030501 18423 log.cc:824] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:28.031054 18422 tablet_bootstrap.cc:492] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: No bootstrap required, opened a new log
I20260927 02:51:28.031131 18422 ts_tablet_manager.cc:1397] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:28.031181 18423 tablet_bootstrap.cc:492] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4: No bootstrap required, opened a new log
I20260927 02:51:28.031239 18423 ts_tablet_manager.cc:1397] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:28.031272 18426 tablet_bootstrap.cc:492] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: Bootstrap starting.
I20260927 02:51:28.031890 18426 tablet_bootstrap.cc:654] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:28.032439 18426 log.cc:824] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:28.033187 18426 tablet_bootstrap.cc:492] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: No bootstrap required, opened a new log
I20260927 02:51:28.033255 18426 ts_tablet_manager.cc:1397] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:28.033207 18423 raft_consensus.cc:348] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.033360 18423 raft_consensus.cc:374] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:28.033399 18423 raft_consensus.cc:729] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b69f465e9b40f9b1ed4b7c276dc0d4, State: Initialized, Role: FOLLOWER
I20260927 02:51:28.033524 18423 consensus_queue.cc:260] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.033483 18422 raft_consensus.cc:348] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.033631 18422 raft_consensus.cc:374] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:28.033664 18422 raft_consensus.cc:729] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2b0cf5cc904e58807412c22ae4002b, State: Initialized, Role: FOLLOWER
I20260927 02:51:28.033756 18423 ts_tablet_manager.cc:1428] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 02:51:28.033776 18422 consensus_queue.cc:260] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.033866 18146 heartbeater.cc:503] Master 127.0.98.62:38219 was elected leader, sending a full tablet report...
I20260927 02:51:28.034014 18422 ts_tablet_manager.cc:1428] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:28.034049 18408 heartbeater.cc:503] Master 127.0.98.62:38219 was elected leader, sending a full tablet report...
I20260927 02:51:28.035429 18426 raft_consensus.cc:348] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.035548 18426 raft_consensus.cc:374] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:28.035567 18426 raft_consensus.cc:729] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd60c3bb32a3460dabbf80eeaaa11332, State: Initialized, Role: FOLLOWER
I20260927 02:51:28.035676 18426 consensus_queue.cc:260] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.035894 18426 ts_tablet_manager.cc:1428] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:28.035939 18277 heartbeater.cc:503] Master 127.0.98.62:38219 was elected leader, sending a full tablet report...
W20260927 02:51:28.106159 18278 tablet.cc:2369] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:51:28.226048 18147 tablet.cc:2369] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:51:28.248603 18409 tablet.cc:2369] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:51:28.249444 18429 raft_consensus.cc:482] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:51:28.249523 18429 raft_consensus.cc:504] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.249872 18429 leader_election.cc:288] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48b69f465e9b40f9b1ed4b7c276dc0d4 (127.0.98.1:43887), dd60c3bb32a3460dabbf80eeaaa11332 (127.0.98.2:46485)
I20260927 02:51:28.253517 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.250217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36441 (local address 127.0.98.2:46485)
0927 02:51:28.250270 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:28.250272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:28.250362 (+ 90us) server_negotiation.cc:407] Connection header received
0927 02:51:28.250406 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:28.250407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:28.250415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:28.250467 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:28.250586 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.251303 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.252626 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.253180 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.253250 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.253259 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.253412 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.253448 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:51:28.253517 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.250143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54831 (local address 127.0.98.1:43887)
0927 02:51:28.250191 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:28.250194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:28.250203 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:28.250236 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:28.250237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:28.250248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:28.250317 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:28.250464 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.251243 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.252649 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.253180 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.253249 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.253259 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.253412 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.253445 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:51:28.253563 18410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.250030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43887 (local address 127.0.98.3:54831)
0927 02:51:28.250076 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:28.250087 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:28.250130 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:28.250325 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:28.250326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:28.250328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:28.250398 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.250400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.251309 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.251310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.252545 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.252547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.253255 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.253256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.253343 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.253348 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:51:28.253401 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:51:28.253446 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":6}
I20260927 02:51:28.253569 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.250128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46485 (local address 127.0.98.3:36441)
0927 02:51:28.250297 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:28.250306 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:28.250356 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:28.250477 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:28.250478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:28.250479 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:28.250532 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.250533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.251366 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.251367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.252545 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:28.252547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.253269 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.253270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:28.253344 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.253348 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:28.253401 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 02:51:28.253446 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":36,"thread_start_us":104,"threads_started":1}
I20260927 02:51:28.253921 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a25cd0de0394cf1ac5eb45f1c173944" candidate_uuid: "9c2b0cf5cc904e58807412c22ae4002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" is_pre_election: true
I20260927 02:51:28.253922 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a25cd0de0394cf1ac5eb45f1c173944" candidate_uuid: "9c2b0cf5cc904e58807412c22ae4002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" is_pre_election: true
I20260927 02:51:28.254058 18100 raft_consensus.cc:2454] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c2b0cf5cc904e58807412c22ae4002b in term 0.
I20260927 02:51:28.254060 18231 raft_consensus.cc:2454] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c2b0cf5cc904e58807412c22ae4002b in term 0.
I20260927 02:51:28.254281 18298 leader_election.cc:302] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b69f465e9b40f9b1ed4b7c276dc0d4, 9c2b0cf5cc904e58807412c22ae4002b; no voters:
I20260927 02:51:28.254391 18429 raft_consensus.cc:2781] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:51:28.254443 18429 raft_consensus.cc:482] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:51:28.254467 18429 raft_consensus.cc:3037] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:28.255157 18429 raft_consensus.cc:504] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.255256 18429 leader_election.cc:288] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [CANDIDATE]: Term 1 election: Requested vote from peers 48b69f465e9b40f9b1ed4b7c276dc0d4 (127.0.98.1:43887), dd60c3bb32a3460dabbf80eeaaa11332 (127.0.98.2:46485)
I20260927 02:51:28.255411 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a25cd0de0394cf1ac5eb45f1c173944" candidate_uuid: "9c2b0cf5cc904e58807412c22ae4002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4"
I20260927 02:51:28.255422 18231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a25cd0de0394cf1ac5eb45f1c173944" candidate_uuid: "9c2b0cf5cc904e58807412c22ae4002b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd60c3bb32a3460dabbf80eeaaa11332"
I20260927 02:51:28.255470 18100 raft_consensus.cc:3037] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:28.255497 18231 raft_consensus.cc:3037] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:28.256145 18100 raft_consensus.cc:2454] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c2b0cf5cc904e58807412c22ae4002b in term 1.
I20260927 02:51:28.256230 18231 raft_consensus.cc:2454] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c2b0cf5cc904e58807412c22ae4002b in term 1.
I20260927 02:51:28.256304 18298 leader_election.cc:302] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b69f465e9b40f9b1ed4b7c276dc0d4, 9c2b0cf5cc904e58807412c22ae4002b; no voters:
I20260927 02:51:28.256392 18429 raft_consensus.cc:2781] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:28.256536 18429 raft_consensus.cc:686] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 1 LEADER]: Becoming Leader. State: Replica: 9c2b0cf5cc904e58807412c22ae4002b, State: Running, Role: LEADER
I20260927 02:51:28.256614 18429 consensus_queue.cc:237] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } }
I20260927 02:51:28.257442 17958 catalog_manager.cc:5495] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c2b0cf5cc904e58807412c22ae4002b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9c2b0cf5cc904e58807412c22ae4002b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c2b0cf5cc904e58807412c22ae4002b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 } health_report { overall_health: UNKNOWN } } }
I20260927 02:51:28.306870 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.304675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.0.98.3:42211)
0927 02:51:28.304728 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:28.304730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:28.304739 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:28.304777 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:28.304778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:28.304796 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:28.304853 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:28.304953 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.305416 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.306168 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:28.306364 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:28.306437 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:28.306548 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:28.306649 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:28.306710 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:28.306726 (+ 16us) server_negotiation.cc:299] Negotiation successful
0927 02:51:28.306742 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:51:28.308650 18100 raft_consensus.cc:1261] T 9a25cd0de0394cf1ac5eb45f1c173944 P 48b69f465e9b40f9b1ed4b7c276dc0d4 [term 1 FOLLOWER]: Refusing update from remote peer 9c2b0cf5cc904e58807412c22ae4002b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:28.308673 18231 raft_consensus.cc:1261] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 1 FOLLOWER]: Refusing update from remote peer 9c2b0cf5cc904e58807412c22ae4002b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:28.308983 18429 consensus_queue.cc:1035] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [LEADER]: Connected to new peer: Peer: permanent_uuid: "48b69f465e9b40f9b1ed4b7c276dc0d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:28.309116 18432 consensus_queue.cc:1035] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd60c3bb32a3460dabbf80eeaaa11332" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:28.311183 18435 mvcc.cc:205] Tried to move back new op lower bound from 7333795792109731840 to 7333795791899770880. Current Snapshot: MvccSnapshot[applied={T|T < 7333795792109731840}]
I20260927 02:51:28.311352 18436 mvcc.cc:205] Tried to move back new op lower bound from 7333795792109731840 to 7333795791899770880. Current Snapshot: MvccSnapshot[applied={T|T < 7333795792109731840}]
I20260927 02:51:28.313388 17958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:28.313463 17958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:28.314123 17958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ff438f75c49490aa5442f66386b40a8: Sending DeleteTablet for 3 replicas of tablet 9a25cd0de0394cf1ac5eb45f1c173944
I20260927 02:51:28.314433 18342 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a25cd0de0394cf1ac5eb45f1c173944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:28 UTC) from {username='slave'} at 127.0.0.1:50024
I20260927 02:51:28.314484 18211 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a25cd0de0394cf1ac5eb45f1c173944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:28 UTC) from {username='slave'} at 127.0.0.1:59556
I20260927 02:51:28.314662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18018
I20260927 02:51:28.314677 18080 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a25cd0de0394cf1ac5eb45f1c173944 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:28 UTC) from {username='slave'} at 127.0.0.1:46424
I20260927 02:51:28.317198 18444 tablet_replica.cc:330] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: stopping tablet replica
I20260927 02:51:28.317299 18444 raft_consensus.cc:2229] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:51:28.317380 18444 raft_consensus.cc:2258] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:28.317773 18444 ts_tablet_manager.cc:1905] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:28.318956 18445 tablet_replica.cc:330] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: stopping tablet replica
I20260927 02:51:28.319115 18445 raft_consensus.cc:2229] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:28.319278 18445 raft_consensus.cc:2258] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:28.319438 18444 ts_tablet_manager.cc:1918] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:28.319543 18444 log.cc:1196] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/9a25cd0de0394cf1ac5eb45f1c173944
I20260927 02:51:28.319860 18444 ts_tablet_manager.cc:1939] T 9a25cd0de0394cf1ac5eb45f1c173944 P 9c2b0cf5cc904e58807412c22ae4002b: Deleting consensus metadata
I20260927 02:51:28.319860 18445 ts_tablet_manager.cc:1905] T 9a25cd0de0394cf1ac5eb45f1c173944 P dd60c3bb32a3460dabbf80eeaaa11332: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:51:28.319912 17945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43887 (error 108)
I20260927 02:51:28.320338 17942 catalog_manager.cc:4841] TS 9c2b0cf5cc904e58807412c22ae4002b (127.0.98.3:42211): tablet 9a25cd0de0394cf1ac5eb45f1c173944 (table test-table [id=d15cc5e7ef264eeda1ed6e15f9e111b4]) successfully deleted
I20260927 02:51:28.320379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18149
I20260927 02:51:28.321714 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.321554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43887 (local address 127.0.0.1:46434)
0927 02:51:28.321614 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:28.321666 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 02:51:28.321866 17945 catalog_manager.cc:4584] TS 48b69f465e9b40f9b1ed4b7c276dc0d4 (127.0.98.1:43887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a25cd0de0394cf1ac5eb45f1c173944: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43887: connect: Connection refused (error 111)
I20260927 02:51:28.324635 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:28.324525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46485 (local address 127.0.0.1:59564)
0927 02:51:28.324574 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:28.324604 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:51:28.325203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18280
I20260927 02:51:28.329945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 17927
2026-09-27T02:51:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (904 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:28 dist-test-slave-f4rg krb5kdc[18449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:28 dist-test-slave-f4rg krb5kdc[18449](info): set up 2 sockets
Sep 27 02:51:28 dist-test-slave-f4rg krb5kdc[18449](info): commencing operation
krb5kdc: starting...
W20260927 02:51:29.270347 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:29.381171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:29.458336 18462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:29.458544 18462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:29.458564 18462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:29.459692 18462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:29.459730 18462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:29.459744 18462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:29.459755 18462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:29.459774 18462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:29.461181 18462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:29.461385 18462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:29.461560 18462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:29.466157 18470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.466153 18468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.466161 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:29.466382 18462 server_base.cc:1034] running on GCE node
I20260927 02:51:29.466675 18462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:29.466977 18462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:29.468129 18462 hybrid_clock.cc:648] HybridClock initialized: now 1790477489468107 us; error 29 us; skew 500 ppm
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:29.470765 18462 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:29.471254 18462 webserver.cc:460] Webserver started at http://127.0.98.62:40407/ using document root <none> and password file <none>
I20260927 02:51:29.471470 18462 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:29.471517 18462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:29.471642 18462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:29.472646 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "4819d97744f347998e75b07a670dc966"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "6c405a75ba5c5ea5f8c1d3c0f38ddd2a"
server_key_iv: "23e30d2eef232c2dc45ea9335107411d"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.472965 18462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "4819d97744f347998e75b07a670dc966"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "6c405a75ba5c5ea5f8c1d3c0f38ddd2a"
server_key_iv: "23e30d2eef232c2dc45ea9335107411d"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.474159 18462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:29.474795 18477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.474946 18462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:29.475071 18462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "4819d97744f347998e75b07a670dc966"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "6c405a75ba5c5ea5f8c1d3c0f38ddd2a"
server_key_iv: "23e30d2eef232c2dc45ea9335107411d"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.475137 18462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:29.491418 18462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:29.493108 18462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:29.493216 18462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:29.496809 18462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36915
I20260927 02:51:29.496853 18529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36915 every 8 connection(s)
I20260927 02:51:29.497174 18462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:29.497973 18530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:29.500159 18530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966: Bootstrap starting.
I20260927 02:51:29.500681 18530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:29.500926 18530 log.cc:824] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:29.501564 18530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966: No bootstrap required, opened a new log
I20260927 02:51:29.503166 18530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4819d97744f347998e75b07a670dc966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260927 02:51:29.503301 18530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:29.503321 18530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4819d97744f347998e75b07a670dc966, State: Initialized, Role: FOLLOWER
I20260927 02:51:29.503445 18530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4819d97744f347998e75b07a670dc966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260927 02:51:29.503509 18530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:29.503531 18530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:29.503577 18530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:29.504058 18530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4819d97744f347998e75b07a670dc966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260927 02:51:29.504153 18530 leader_election.cc:302] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4819d97744f347998e75b07a670dc966; no voters:
I20260927 02:51:29.504307 18530 leader_election.cc:288] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:29.504345 18533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:29.504503 18533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [term 1 LEADER]: Becoming Leader. State: Replica: 4819d97744f347998e75b07a670dc966, State: Running, Role: LEADER
I20260927 02:51:29.504520 18530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:29.504591 18533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4819d97744f347998e75b07a670dc966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } }
I20260927 02:51:29.505136 18535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4819d97744f347998e75b07a670dc966. Latest consensus state: current_term: 1 leader_uuid: "4819d97744f347998e75b07a670dc966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4819d97744f347998e75b07a670dc966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260927 02:51:29.505226 18535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:29.505354 18534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4819d97744f347998e75b07a670dc966" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4819d97744f347998e75b07a670dc966" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36915 } } }
I20260927 02:51:29.505476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18462
I20260927 02:51:29.505493 18534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:29.505589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:29.505771 392 external_mini_cluster.cc:1428] Setting key 466a705f9076748fd2ebf9ead9a7f700
W20260927 02:51:29.506155 18549 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:51:29.506242 18549 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:29.506299 18542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:29.506482 18542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:29.507948 18542 catalog_manager.cc:1347] Generated new cluster ID: 3baeed39d0a24122ba1e1630f4c2663d
I20260927 02:51:29.508000 18542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:29.515872 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.506220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.0.98.62:36915)
0927 02:51:29.507646 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:29.507655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:29.507679 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:51:29.507768 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:29.507774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:29.507814 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:29.507936 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:29.508664 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.509582 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.510309 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.510712 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.510833 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.512920 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:29.512925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:29.512925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:29.512935 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:29.514433 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.514744 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.514747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.514747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.514877 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.515065 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.515066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.515066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.515304 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:29.515440 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:29.515546 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 02:51:29.515680 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368,"thread_start_us":66,"threads_started":1}
I20260927 02:51:29.522295 18542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:29.522784 18542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:29.528617 18542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966: Generated new TSK 0
I20260927 02:51:29.528822 18542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:51:29.559548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:29.638315 18556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:29.638509 18556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:29.638531 18556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:29.639691 18556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:29.639734 18556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:29.639761 18556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:51:29.641218 18556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:29.641445 18556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:29.641662 18556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:51:29.646377 18556 server_base.cc:1034] running on GCE node
W20260927 02:51:29.646389 18562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.646391 18561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.646490 18564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:29.646786 18556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:29.647027 18556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:29.648164 18556 hybrid_clock.cc:648] HybridClock initialized: now 1790477489648149 us; error 24 us; skew 500 ppm
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:29.650454 18556 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:51:29.650888 18556 webserver.cc:460] Webserver started at http://127.0.98.1:35033/ using document root <none> and password file <none>
I20260927 02:51:29.651093 18556 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:29.651140 18556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:29.651270 18556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:29.652274 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "cdc42df41061d1b2069847d075539180"
server_key_iv: "e24bdf1d7bbe43daeae20ca4517fbc85"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.652591 18556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "cdc42df41061d1b2069847d075539180"
server_key_iv: "e24bdf1d7bbe43daeae20ca4517fbc85"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.653785 18556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:51:29.654414 18571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.654552 18556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:29.654628 18556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "cdc42df41061d1b2069847d075539180"
server_key_iv: "e24bdf1d7bbe43daeae20ca4517fbc85"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.654691 18556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:29.668951 18556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:29.670553 18556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:29.670679 18556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:29.670902 18556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:29.671190 18556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:29.671223 18556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.671250 18556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:29.671273 18556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.676647 18556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38325
I20260927 02:51:29.676721 18684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38325 every 8 connection(s)
I20260927 02:51:29.676987 18556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:29.684022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18556
I20260927 02:51:29.684139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:51:29.684384 392 external_mini_cluster.cc:1428] Setting key e7ee07de3a4bfb982cb26dfa5f79bbaa
I20260927 02:51:29.685987 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.677669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48563 (local address 127.0.98.62:36915)
0927 02:51:29.677713 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:29.677715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:29.677890 (+ 175us) server_negotiation.cc:407] Connection header received
0927 02:51:29.678841 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:29.678842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:29.678858 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:29.678913 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:29.679584 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.680101 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.681060 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.681274 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.681332 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.683284 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:29.683285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:29.683286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:29.683294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:29.684743 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.685114 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.685115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.685115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.685174 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.685386 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.685387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.685388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.685504 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:29.685560 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:29.685818 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:51:29.685902 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:51:29.686180 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.677660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.1:48563)
0927 02:51:29.677789 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:29.677824 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:29.678731 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:29.678970 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:29.678973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:29.679228 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:29.679418 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:29.679422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.680183 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.680183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:29.680977 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:29.680979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.681338 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.681339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:29.681432 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.681873 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:29.681881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:29.683197 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:29.684822 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:29.684831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:29.684834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:29.685050 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:29.685227 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:29.685227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:29.685228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:29.685332 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:29.685557 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:29.685563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:29.685703 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 02:51:29.685790 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 02:51:29.685943 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
I20260927 02:51:29.686707 18685 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260927 02:51:29.686839 18685 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:29.687036 18685 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260927 02:51:29.687525 18494 ts_manager.cc:194] Registered new tserver with Master: 38b9dc5ecb4848e8bc71447acf3e1a4a (127.0.98.1:38325)
I20260927 02:51:29.688172 18494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48563
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:51:29.707515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:29.784878 18690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:29.785075 18690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:29.785096 18690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:29.786212 18690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:29.786245 18690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:29.786269 18690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:51:29.787712 18690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:29.787906 18690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:29.788085 18690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:29.792748 18696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.792748 18698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.792744 18695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:29.792965 18690 server_base.cc:1034] running on GCE node
I20260927 02:51:29.793157 18690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:29.793387 18690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:29.794519 18690 hybrid_clock.cc:648] HybridClock initialized: now 1790477489794504 us; error 33 us; skew 500 ppm
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:29.796744 18690 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:51:29.797173 18690 webserver.cc:460] Webserver started at http://127.0.98.2:44603/ using document root <none> and password file <none>
I20260927 02:51:29.797379 18690 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:29.797426 18690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:29.797544 18690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:29.798594 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "7711ff8e053744bd98a16bbc0b9e8a05"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "bf4d3d17a750c76f4942f72fc785e5fe"
server_key_iv: "6a95c7fb15bb3e8e773f407217d5d157"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.798969 18690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7711ff8e053744bd98a16bbc0b9e8a05"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "bf4d3d17a750c76f4942f72fc785e5fe"
server_key_iv: "6a95c7fb15bb3e8e773f407217d5d157"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.800194 18690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:29.800876 18705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.801029 18690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:29.801105 18690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "7711ff8e053744bd98a16bbc0b9e8a05"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "bf4d3d17a750c76f4942f72fc785e5fe"
server_key_iv: "6a95c7fb15bb3e8e773f407217d5d157"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.801174 18690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:29.823402 18690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:29.825192 18690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:29.825330 18690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:29.825542 18690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:29.825841 18690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:29.825874 18690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.825903 18690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:29.825920 18690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.831180 18690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36705
I20260927 02:51:29.831233 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36705 every 8 connection(s)
I20260927 02:51:29.831507 18690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:51:29.831929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18690
I20260927 02:51:29.832046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:51:29.832263 392 external_mini_cluster.cc:1428] Setting key 9567173d8d7aed456368dd05edafcfd4
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:51:29.841008 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.832168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52091 (local address 127.0.98.62:36915)
0927 02:51:29.832229 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:29.832232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:29.832401 (+ 169us) server_negotiation.cc:407] Connection header received
0927 02:51:29.833403 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:29.833405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:29.833417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:29.833474 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:29.834084 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.834620 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.836013 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.836253 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.836305 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.838639 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:29.838640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:29.838641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:29.838648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:29.839911 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.840243 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.840243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.840244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.840287 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.840456 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.840457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.840457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.840555 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:29.840605 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:29.840871 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 02:51:29.840942 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:51:29.841111 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.832114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.2:52091)
0927 02:51:29.832285 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:29.832325 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:29.833300 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:29.833534 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:29.833537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:29.833786 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:29.833990 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:29.833996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.834699 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.834700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:29.835937 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:29.835941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.836328 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.836329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:29.836477 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.836982 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:29.836993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:29.838536 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:29.839980 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:29.839987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:29.839989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:29.840187 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:29.840330 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:29.840331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:29.840331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:29.840414 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:29.840601 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:29.840604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:29.840751 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 02:51:29.840832 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:51:29.840942 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260927 02:51:29.841526 18819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260927 02:51:29.841653 18819 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:29.841811 18819 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260927 02:51:29.842140 18494 ts_manager.cc:194] Registered new tserver with Master: 7711ff8e053744bd98a16bbc0b9e8a05 (127.0.98.2:36705)
I20260927 02:51:29.842479 18494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:51:29.857471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:29.938512 18824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:29.938720 18824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:29.938751 18824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:29.939915 18824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:29.939954 18824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:29.939989 18824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:51:29.941431 18824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:29.941658 18824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:29.941850 18824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:29.946380 18830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.946442 18832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:29.946444 18829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:29.946775 18824 server_base.cc:1034] running on GCE node
I20260927 02:51:29.947006 18824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:29.947222 18824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:29.948381 18824 hybrid_clock.cc:648] HybridClock initialized: now 1790477489948370 us; error 25 us; skew 500 ppm
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:29.950711 18824 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:51:29.951141 18824 webserver.cc:460] Webserver started at http://127.0.98.3:33385/ using document root <none> and password file <none>
I20260927 02:51:29.951340 18824 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:29.951403 18824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:29.951531 18824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:29.952675 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "324deac6e93541dfb81043c593a7d652"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "ad9b9416bf5ada63cdfbbb89812229c0"
server_key_iv: "2f328cb8d75ce09e73d66e137c74fb71"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.953015 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "324deac6e93541dfb81043c593a7d652"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "ad9b9416bf5ada63cdfbbb89812229c0"
server_key_iv: "2f328cb8d75ce09e73d66e137c74fb71"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.954269 18824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:51:29.954980 18839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.955135 18824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:29.955209 18824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "324deac6e93541dfb81043c593a7d652"
format_stamp: "Formatted at 2026-09-27 02:51:29 on dist-test-slave-f4rg"
server_key: "ad9b9416bf5ada63cdfbbb89812229c0"
server_key_iv: "2f328cb8d75ce09e73d66e137c74fb71"
server_key_version: "encryptionkey@0"
I20260927 02:51:29.955276 18824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:29.969765 18824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:29.971722 18824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:29.971861 18824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:29.972070 18824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:29.972357 18824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:29.972389 18824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.972427 18824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:29.972458 18824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:29.977968 18824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38967
I20260927 02:51:29.978029 18952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38967 every 8 connection(s)
I20260927 02:51:29.978384 18824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:51:29.981887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18824
I20260927 02:51:29.981988 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:51:29.982170 392 external_mini_cluster.cc:1428] Setting key 87b1be3c9570f049e7d191a3ab0803ea
Sep 27 02:51:29 dist-test-slave-f4rg krb5kdc[18449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:29.987247 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.979085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41851 (local address 127.0.98.62:36915)
0927 02:51:29.979129 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:29.979131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:29.979335 (+ 204us) server_negotiation.cc:407] Connection header received
0927 02:51:29.980240 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:29.980241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:29.980252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:29.980306 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:29.980894 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.981381 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.982574 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.982840 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.982901 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.984975 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:29.984976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:29.984977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:29.984984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:29.986032 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.986379 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.986380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.986381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.986436 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.986643 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.986643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.986644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.986752 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:29.986826 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:29.987071 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 02:51:29.987149 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:51:29.987397 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.979066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36915 (local address 127.0.98.3:41851)
0927 02:51:29.979216 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:29.979257 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:29.980149 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:29.980366 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:29.980369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:29.980629 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:29.980825 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:29.980830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.981478 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.981480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:29.982489 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:29.982492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.982913 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.982914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:29.983059 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.983561 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:29.983569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:29.984882 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:29.986103 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:29.986109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:29.986111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:29.986315 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:29.986489 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:29.986489 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:29.986490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:29.986590 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:29.986819 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:29.986825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:29.986957 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 02:51:29.987043 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 02:51:29.987191 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260927 02:51:29.987807 18953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36915
I20260927 02:51:29.987895 18953 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:29.988031 18953 heartbeater.cc:511] Master 127.0.98.62:36915 requested a full tablet report, sending...
I20260927 02:51:29.988353 18494 ts_manager.cc:194] Registered new tserver with Master: 324deac6e93541dfb81043c593a7d652 (127.0.98.3:38967)
I20260927 02:51:29.988724 18494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41851
I20260927 02:51:29.993842 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:51:30.000532 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:29.995229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.0.98.62:36915)
0927 02:51:29.995270 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:29.995272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:29.995371 (+ 99us) server_negotiation.cc:407] Connection header received
0927 02:51:29.995422 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:29.995422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:29.995431 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:29.995475 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:29.995947 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.996426 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.997116 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:29.997295 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:29.997338 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:29.998519 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:29.998520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:29.998520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:29.998525 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:29.999611 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:29.999948 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:29.999948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:29.999949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:29.999992 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:30.000139 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:30.000139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:30.000139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:30.000254 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:30.000323 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.000426 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.000469 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:30.001967 18494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:51:30.003535 18494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:51:30.009656 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.005860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.0.98.3:38967)
0927 02:51:30.006110 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.006114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.006122 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:30.006155 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.006157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.006172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.006237 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.006474 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.007267 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.008571 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.009269 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.009339 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.009396 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.009524 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.009563 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":62,"threads_started":1}
I20260927 02:51:30.009650 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.005585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38967 (local address 127.0.0.1:54930)
0927 02:51:30.005751 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.005924 (+ 173us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.005962 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.006320 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.006321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.006323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.006396 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.006399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.007355 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.007356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.008501 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.008504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.009349 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.009350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.009424 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.009436 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.009505 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.009564 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":54,"threads_started":1}
I20260927 02:51:30.009651 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.005587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38325 (local address 127.0.0.1:39386)
0927 02:51:30.005856 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.005886 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.005953 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.006235 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.006239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.006241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.006401 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.006404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.007770 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.007771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.008713 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.008715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.009455 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.009455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.009509 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.009512 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.009551 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.009581 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":32,"thread_start_us":56,"threads_started":1}
I20260927 02:51:30.010027 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.005722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.2:36705)
0927 02:51:30.006040 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.006044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.006095 (+ 51us) server_negotiation.cc:407] Connection header received
0927 02:51:30.006198 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.006201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.006217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.006279 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.006476 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.007656 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.008861 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.009547 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.009607 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.009656 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.009913 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.009956 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":67,"threads_started":1}
I20260927 02:51:30.010110 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.005826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.0.98.1:38325)
0927 02:51:30.006002 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.006006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.006017 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:51:30.006072 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.006074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.006138 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.006232 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.006484 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.007656 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.008792 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.009305 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.009369 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.009425 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.009690 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.009729 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260927 02:51:30.010170 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.005587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36705 (local address 127.0.0.1:41056)
0927 02:51:30.006042 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.006049 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.006130 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.006362 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.006363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.006365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.006408 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.006409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.007761 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.007762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.008795 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.008798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.009691 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.009692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.009752 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.009756 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.009802 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.009841 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":26,"thread_start_us":83,"threads_started":1}
I20260927 02:51:30.010591 18887 tablet_service.cc:1467] Processing CreateTablet for tablet 055fb669f01f424daa6aa1da72d36283 (DEFAULT_TABLE table=test-table [id=2229e0369b86476e871d56bcfd93ff7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:30.010776 18619 tablet_service.cc:1467] Processing CreateTablet for tablet 055fb669f01f424daa6aa1da72d36283 (DEFAULT_TABLE table=test-table [id=2229e0369b86476e871d56bcfd93ff7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:30.010776 18753 tablet_service.cc:1467] Processing CreateTablet for tablet 055fb669f01f424daa6aa1da72d36283 (DEFAULT_TABLE table=test-table [id=2229e0369b86476e871d56bcfd93ff7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:30.010969 18887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055fb669f01f424daa6aa1da72d36283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:30.011103 18619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055fb669f01f424daa6aa1da72d36283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:30.011125 18753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055fb669f01f424daa6aa1da72d36283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:30.013442 18967 tablet_bootstrap.cc:492] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: Bootstrap starting.
I20260927 02:51:30.014065 18967 tablet_bootstrap.cc:654] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:30.014071 18968 tablet_bootstrap.cc:492] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a: Bootstrap starting.
I20260927 02:51:30.014664 18968 tablet_bootstrap.cc:654] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:30.014675 18967 log.cc:824] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:30.014945 18969 tablet_bootstrap.cc:492] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Bootstrap starting.
I20260927 02:51:30.015190 18968 log.cc:824] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:30.015468 18967 tablet_bootstrap.cc:492] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: No bootstrap required, opened a new log
I20260927 02:51:30.015534 18969 tablet_bootstrap.cc:654] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:30.015528 18967 ts_tablet_manager.cc:1397] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 02:51:30.015910 18968 tablet_bootstrap.cc:492] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a: No bootstrap required, opened a new log
I20260927 02:51:30.016019 18968 ts_tablet_manager.cc:1397] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:51:30.016059 18969 log.cc:824] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:30.016645 18969 tablet_bootstrap.cc:492] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: No bootstrap required, opened a new log
I20260927 02:51:30.016693 18969 ts_tablet_manager.cc:1397] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:30.017411 18967 raft_consensus.cc:348] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.017608 18967 raft_consensus.cc:374] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:30.017644 18967 raft_consensus.cc:729] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7711ff8e053744bd98a16bbc0b9e8a05, State: Initialized, Role: FOLLOWER
I20260927 02:51:30.017786 18967 consensus_queue.cc:260] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.018044 18967 ts_tablet_manager.cc:1428] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 02:51:30.017956 18968 raft_consensus.cc:348] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.018116 18968 raft_consensus.cc:374] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:30.018160 18819 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
I20260927 02:51:30.018157 18968 raft_consensus.cc:729] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b9dc5ecb4848e8bc71447acf3e1a4a, State: Initialized, Role: FOLLOWER
I20260927 02:51:30.018260 18968 consensus_queue.cc:260] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.018308 18969 raft_consensus.cc:348] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.018466 18969 raft_consensus.cc:374] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:30.018502 18969 raft_consensus.cc:729] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324deac6e93541dfb81043c593a7d652, State: Initialized, Role: FOLLOWER
I20260927 02:51:30.018513 18968 ts_tablet_manager.cc:1428] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 02:51:30.018610 18685 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
I20260927 02:51:30.018608 18969 consensus_queue.cc:260] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.018877 18969 ts_tablet_manager.cc:1428] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:51:30.018989 18953 heartbeater.cc:503] Master 127.0.98.62:36915 was elected leader, sending a full tablet report...
W20260927 02:51:30.082011 18820 tablet.cc:2369] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:51:30.135182 18975 raft_consensus.cc:482] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:51:30.135316 18975 raft_consensus.cc:504] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.135730 18975 leader_election.cc:288] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7711ff8e053744bd98a16bbc0b9e8a05 (127.0.98.2:36705), 38b9dc5ecb4848e8bc71447acf3e1a4a (127.0.98.1:38325)
I20260927 02:51:30.138389 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.135999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56779 (local address 127.0.98.2:36705)
0927 02:51:30.136069 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.136071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.136098 (+ 27us) server_negotiation.cc:407] Connection header received
0927 02:51:30.136128 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.136129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.136142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.136186 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.136330 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.136789 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.137712 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.138077 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.138143 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.138155 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.138290 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.138328 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:51:30.138389 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.135891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36705 (local address 127.0.98.3:56779)
0927 02:51:30.135960 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.135972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.136011 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.136206 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.136207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.136208 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.136273 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.136275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.136864 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.136865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.137625 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.137628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.138158 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.138159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.138231 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.138234 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.138277 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.138315 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 02:51:30.138487 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.136166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42377 (local address 127.0.98.1:38325)
0927 02:51:30.136213 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.136215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.136223 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:30.136245 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.136246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.136254 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.136297 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.136446 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.136927 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.137824 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.138172 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.138228 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.138238 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.138378 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.138420 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:51:30.138499 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.135928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38325 (local address 127.0.98.3:42377)
0927 02:51:30.136077 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.136087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.136142 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.136315 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.136315 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.136317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.136385 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.136386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.137011 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.137012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.137740 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.137742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.138246 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.138247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.138317 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.138320 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.138362 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.138398 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260927 02:51:30.138756 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055fb669f01f424daa6aa1da72d36283" candidate_uuid: "324deac6e93541dfb81043c593a7d652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" is_pre_election: true
I20260927 02:51:30.138752 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055fb669f01f424daa6aa1da72d36283" candidate_uuid: "324deac6e93541dfb81043c593a7d652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" is_pre_election: true
I20260927 02:51:30.138931 18773 raft_consensus.cc:2454] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324deac6e93541dfb81043c593a7d652 in term 0.
I20260927 02:51:30.138932 18639 raft_consensus.cc:2454] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324deac6e93541dfb81043c593a7d652 in term 0.
I20260927 02:51:30.139155 18841 leader_election.cc:302] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324deac6e93541dfb81043c593a7d652, 38b9dc5ecb4848e8bc71447acf3e1a4a; no voters:
I20260927 02:51:30.139276 18975 raft_consensus.cc:2781] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:51:30.139330 18975 raft_consensus.cc:482] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:51:30.139353 18975 raft_consensus.cc:3037] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:30.139989 18975 raft_consensus.cc:504] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.140103 18975 leader_election.cc:288] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [CANDIDATE]: Term 1 election: Requested vote from peers 7711ff8e053744bd98a16bbc0b9e8a05 (127.0.98.2:36705), 38b9dc5ecb4848e8bc71447acf3e1a4a (127.0.98.1:38325)
I20260927 02:51:30.140266 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055fb669f01f424daa6aa1da72d36283" candidate_uuid: "324deac6e93541dfb81043c593a7d652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7711ff8e053744bd98a16bbc0b9e8a05"
I20260927 02:51:30.140282 18639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055fb669f01f424daa6aa1da72d36283" candidate_uuid: "324deac6e93541dfb81043c593a7d652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a"
I20260927 02:51:30.140354 18773 raft_consensus.cc:3037] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:30.140355 18639 raft_consensus.cc:3037] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:30.141204 18773 raft_consensus.cc:2454] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324deac6e93541dfb81043c593a7d652 in term 1.
I20260927 02:51:30.141204 18639 raft_consensus.cc:2454] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324deac6e93541dfb81043c593a7d652 in term 1.
I20260927 02:51:30.141352 18841 leader_election.cc:302] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 324deac6e93541dfb81043c593a7d652, 38b9dc5ecb4848e8bc71447acf3e1a4a; no voters:
I20260927 02:51:30.141458 18975 raft_consensus.cc:2781] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:30.141608 18975 raft_consensus.cc:686] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 1 LEADER]: Becoming Leader. State: Replica: 324deac6e93541dfb81043c593a7d652, State: Running, Role: LEADER
I20260927 02:51:30.141707 18975 consensus_queue.cc:237] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } }
I20260927 02:51:30.142611 18494 catalog_manager.cc:5495] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 324deac6e93541dfb81043c593a7d652 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "324deac6e93541dfb81043c593a7d652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "324deac6e93541dfb81043c593a7d652" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38967 } health_report { overall_health: HEALTHY } } }
I20260927 02:51:30.157006 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.154670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.0.98.3:38967)
0927 02:51:30.154732 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.154734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.154743 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:30.154841 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.154842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.154860 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.154917 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:30.155026 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.155498 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.156207 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.156391 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.156469 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.156618 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:30.156728 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:30.156830 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.156900 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.156908 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:51:30.158849 18773 raft_consensus.cc:1261] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 1 FOLLOWER]: Refusing update from remote peer 324deac6e93541dfb81043c593a7d652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:30.158849 18639 raft_consensus.cc:1261] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a [term 1 FOLLOWER]: Refusing update from remote peer 324deac6e93541dfb81043c593a7d652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:30.159188 18975 consensus_queue.cc:1035] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b9dc5ecb4848e8bc71447acf3e1a4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:30.159334 18977 consensus_queue.cc:1035] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7711ff8e053744bd98a16bbc0b9e8a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:30.161834 18981 mvcc.cc:205] Tried to move back new op lower bound from 7333795799687888896 to 7333795799621095424. Current Snapshot: MvccSnapshot[applied={T|T < 7333795799687888896}]
I20260927 02:51:30.163630 18493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:30.163738 18493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:30.164449 18493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4819d97744f347998e75b07a670dc966: Sending DeleteTablet for 3 replicas of tablet 055fb669f01f424daa6aa1da72d36283
I20260927 02:51:30.164783 18753 tablet_service.cc:1514] Processing DeleteTablet for tablet 055fb669f01f424daa6aa1da72d36283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41056
I20260927 02:51:30.164834 18887 tablet_service.cc:1514] Processing DeleteTablet for tablet 055fb669f01f424daa6aa1da72d36283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54930
I20260927 02:51:30.165051 18991 tablet_replica.cc:330] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: stopping tablet replica
I20260927 02:51:30.165046 18619 tablet_service.cc:1514] Processing DeleteTablet for tablet 055fb669f01f424daa6aa1da72d36283 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39386
I20260927 02:51:30.165218 18991 raft_consensus.cc:2229] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:51:30.165238 18992 tablet_replica.cc:330] T 055fb669f01f424daa6aa1da72d36283 P 38b9dc5ecb4848e8bc71447acf3e1a4a: stopping tablet replica
I20260927 02:51:30.165350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18556
I20260927 02:51:30.166587 18991 raft_consensus.cc:2258] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:30.166683 18990 tablet_replica.cc:330] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: stopping tablet replica
I20260927 02:51:30.166782 18990 raft_consensus.cc:2229] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:30.166903 18990 raft_consensus.cc:2258] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:30.167186 18991 ts_tablet_manager.cc:1905] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:30.168779 18991 ts_tablet_manager.cc:1918] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:30.168846 18991 log.cc:1196] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/055fb669f01f424daa6aa1da72d36283
I20260927 02:51:30.169097 18991 ts_tablet_manager.cc:1939] T 055fb669f01f424daa6aa1da72d36283 P 324deac6e93541dfb81043c593a7d652: Deleting consensus metadata
I20260927 02:51:30.169449 18480 catalog_manager.cc:4841] TS 324deac6e93541dfb81043c593a7d652 (127.0.98.3:38967): tablet 055fb669f01f424daa6aa1da72d36283 (table test-table [id=2229e0369b86476e871d56bcfd93ff7c]) successfully deleted
W20260927 02:51:30.169528 18479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38325 (error 108)
I20260927 02:51:30.169932 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.169836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38325 (local address 127.0.0.1:39394)
0927 02:51:30.169875 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.169906 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 02:51:30.170020 18990 ts_tablet_manager.cc:1905] T 055fb669f01f424daa6aa1da72d36283 P 7711ff8e053744bd98a16bbc0b9e8a05: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 02:51:30.170035 18479 catalog_manager.cc:4584] TS 38b9dc5ecb4848e8bc71447acf3e1a4a (127.0.98.1:38325): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 055fb669f01f424daa6aa1da72d36283: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38325: connect: Connection refused (error 111)
I20260927 02:51:30.170320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18690
I20260927 02:51:30.176221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18824
I20260927 02:51:30.176437 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.176320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36705 (local address 127.0.0.1:41066)
0927 02:51:30.176380 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.176404 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:51:30.180702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18462
2026-09-27T02:51:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1851 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T02:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:30Z Disabled control of system clock
I20260927 02:51:30.206049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34505
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:51:30.285194 18997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:30.285400 18997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:30.285419 18997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:30.286571 18997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:30.286609 18997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:30.286621 18997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:30.286634 18997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:30.286645 18997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:30.287997 18997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:30.288198 18997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:30.288429 18997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:51:30.292927 18997 server_base.cc:1034] running on GCE node
W20260927 02:51:30.292879 19002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:30.292850 19005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:30.292850 19003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:30.293238 18997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:30.293486 18997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:30.294615 18997 hybrid_clock.cc:648] HybridClock initialized: now 1790477490294600 us; error 29 us; skew 500 ppm
I20260927 02:51:30.295760 18997 webserver.cc:460] Webserver started at http://127.0.98.62:41777/ using document root <none> and password file <none>
I20260927 02:51:30.295976 18997 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:30.296020 18997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:30.296162 18997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:30.297173 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "abfff5e4e65648b685f1c06f0bac56e4"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "66bea85f08f8f4597c3d4cb75e8c1404"
server_key_iv: "4618a48d39f7ac9b4fdedf906e9a7463"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.297497 18997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "abfff5e4e65648b685f1c06f0bac56e4"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "66bea85f08f8f4597c3d4cb75e8c1404"
server_key_iv: "4618a48d39f7ac9b4fdedf906e9a7463"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.298682 18997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:30.299366 19011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.299508 18997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.299577 18997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "abfff5e4e65648b685f1c06f0bac56e4"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "66bea85f08f8f4597c3d4cb75e8c1404"
server_key_iv: "4618a48d39f7ac9b4fdedf906e9a7463"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.299628 18997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:30.316900 18997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:30.317190 18997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:30.317293 18997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:30.320909 18997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34505
I20260927 02:51:30.320942 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34505 every 8 connection(s)
I20260927 02:51:30.321259 18997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:30.322146 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:30.324306 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4: Bootstrap starting.
I20260927 02:51:30.324812 19064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:30.325037 19064 log.cc:824] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:30.325587 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4: No bootstrap required, opened a new log
I20260927 02:51:30.327136 19064 raft_consensus.cc:348] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfff5e4e65648b685f1c06f0bac56e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34505 } }
I20260927 02:51:30.327270 19064 raft_consensus.cc:374] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:30.327291 19064 raft_consensus.cc:729] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfff5e4e65648b685f1c06f0bac56e4, State: Initialized, Role: FOLLOWER
I20260927 02:51:30.327394 19064 consensus_queue.cc:260] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfff5e4e65648b685f1c06f0bac56e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34505 } }
I20260927 02:51:30.327440 19064 raft_consensus.cc:388] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:30.327471 19064 raft_consensus.cc:482] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:30.327499 19064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:30.327937 19064 raft_consensus.cc:504] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfff5e4e65648b685f1c06f0bac56e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34505 } }
I20260927 02:51:30.328015 19064 leader_election.cc:302] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abfff5e4e65648b685f1c06f0bac56e4; no voters:
I20260927 02:51:30.328130 19064 leader_election.cc:288] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:30.328187 19067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:30.328337 19067 raft_consensus.cc:686] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [term 1 LEADER]: Becoming Leader. State: Replica: abfff5e4e65648b685f1c06f0bac56e4, State: Running, Role: LEADER
I20260927 02:51:30.328346 19064 sys_catalog.cc:564] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:30.328442 19067 consensus_queue.cc:237] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfff5e4e65648b685f1c06f0bac56e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34505 } }
I20260927 02:51:30.328910 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfff5e4e65648b685f1c06f0bac56e4. Latest consensus state: current_term: 1 leader_uuid: "abfff5e4e65648b685f1c06f0bac56e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfff5e4e65648b685f1c06f0bac56e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34505 } } }
I20260927 02:51:30.328989 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:30.329182 19076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:30.329339 19068 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abfff5e4e65648b685f1c06f0bac56e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfff5e4e65648b685f1c06f0bac56e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34505 } } }
I20260927 02:51:30.329442 19068 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:30.329674 19076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:30.329784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 18997
I20260927 02:51:30.329874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:30.330044 392 external_mini_cluster.cc:1428] Setting key 4c94827522d2de735617669d74a63e2e
I20260927 02:51:30.331075 19076 catalog_manager.cc:1347] Generated new cluster ID: 8c4905edcbeb4f57be6594366fafb213
I20260927 02:51:30.331138 19076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:30.335939 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.330537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.0.98.62:34505)
0927 02:51:30.330707 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.330718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.330795 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:51:30.330957 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.330965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.331043 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.331188 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:30.331501 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.332830 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.333902 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.334533 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.334655 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.335253 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:30.335257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:30.335257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:30.335470 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:30.335515 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:30.335530 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:30.335596 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.335657 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.335736 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260927 02:51:30.338035 19076 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:30.338510 19076 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:30.345301 19076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4: Generated new TSK 0
I20260927 02:51:30.345458 19076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:30.350476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34505
--builtin_ntp_servers=127.0.98.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:51:30.428910 19088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:30.429155 19088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:30.429190 19088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:30.430388 19088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:30.430425 19088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:30.430457 19088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:51:30.431810 19088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:30.432034 19088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:30.432307 19088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:30.437011 19096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:30.437094 19094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:30.437004 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:30.438968 19088 server_base.cc:1034] running on GCE node
I20260927 02:51:30.439174 19088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:30.439404 19088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:30.440583 19088 hybrid_clock.cc:648] HybridClock initialized: now 1790477490440564 us; error 31 us; skew 500 ppm
I20260927 02:51:30.446398 19088 webserver.cc:460] Webserver started at http://127.0.98.1:40501/ using document root <none> and password file <none>
I20260927 02:51:30.446633 19088 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:30.446681 19088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:30.446844 19088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:30.447928 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "29d548f6a5d94234bc30a7c6d3d12538"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "52e849e0d2491c88c439f4d017f688cd"
server_key_iv: "214b4e72d0d8242c39ccfb898671af04"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.448264 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29d548f6a5d94234bc30a7c6d3d12538"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "52e849e0d2491c88c439f4d017f688cd"
server_key_iv: "214b4e72d0d8242c39ccfb898671af04"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.449476 19088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 02:51:30.450140 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.450331 19088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:30.450431 19088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "29d548f6a5d94234bc30a7c6d3d12538"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "52e849e0d2491c88c439f4d017f688cd"
server_key_iv: "214b4e72d0d8242c39ccfb898671af04"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.450502 19088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:30.463426 19088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:30.463706 19088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:30.463804 19088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:30.463986 19088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:30.464274 19088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:30.464306 19088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.464327 19088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:30.464339 19088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.469550 19088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45053
I20260927 02:51:30.469619 19215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45053 every 8 connection(s)
I20260927 02:51:30.469915 19088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:51:30.474897 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.470638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33033 (local address 127.0.98.62:34505)
0927 02:51:30.470697 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.470699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.470949 (+ 250us) server_negotiation.cc:407] Connection header received
0927 02:51:30.471438 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.471439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.471451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.471543 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:30.472157 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.472702 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.473623 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.473821 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.473882 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.474619 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:30.474620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:30.474621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:30.474626 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:30.474642 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:30.474646 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:30.474696 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.474757 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.474785 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:51:30.475030 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.470633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34505 (local address 127.0.98.1:33033)
0927 02:51:30.470793 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.470866 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.471352 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.471604 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.471608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.471616 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:30.472096 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.472100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.472764 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.472765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.473531 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.473532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.473894 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.473895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.474003 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.474483 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:30.474484 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:30.474511 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:30.474532 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:30.474701 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:30.474703 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:30.474706 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.474751 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.474855 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260927 02:51:30.475245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 19088
I20260927 02:51:30.475332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:51:30.475473 392 external_mini_cluster.cc:1428] Setting key 78c263caf86336a2ee13defa3ddca2e7
I20260927 02:51:30.476029 19216 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34505
I20260927 02:51:30.476137 19216 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:30.476317 19216 heartbeater.cc:511] Master 127.0.98.62:34505 requested a full tablet report, sending...
I20260927 02:51:30.476414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34505
--builtin_ntp_servers=127.0.98.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:51:30.476788 19028 ts_manager.cc:194] Registered new tserver with Master: 29d548f6a5d94234bc30a7c6d3d12538 (127.0.98.1:45053)
I20260927 02:51:30.477372 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33033
W20260927 02:51:30.554491 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:30.554723 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:30.554752 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:30.555963 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:30.556012 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:30.556046 19219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:51:30.557403 19219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:30.557677 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:30.557966 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:30.562880 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:30.562875 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:30.562934 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:30.563129 19219 server_base.cc:1034] running on GCE node
I20260927 02:51:30.563323 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:30.563575 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:30.564715 19219 hybrid_clock.cc:648] HybridClock initialized: now 1790477490564703 us; error 28 us; skew 500 ppm
I20260927 02:51:30.565943 19219 webserver.cc:460] Webserver started at http://127.0.98.2:37969/ using document root <none> and password file <none>
I20260927 02:51:30.566162 19219 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:30.566208 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:30.566322 19219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:30.567394 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "d1d2b130fbea4717bce1d26805017961"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "01f159c7e9aedaecb0b341468093a021"
server_key_iv: "9603efe6e96f07c33d302ab65d956797"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.567744 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d1d2b130fbea4717bce1d26805017961"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "01f159c7e9aedaecb0b341468093a021"
server_key_iv: "9603efe6e96f07c33d302ab65d956797"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.568912 19219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:30.569599 19233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.569761 19219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:30.569841 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "d1d2b130fbea4717bce1d26805017961"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "01f159c7e9aedaecb0b341468093a021"
server_key_iv: "9603efe6e96f07c33d302ab65d956797"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.569900 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:30.583261 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:30.583535 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:30.583638 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:30.583811 19219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:30.584107 19219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:30.584138 19219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.584168 19219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:30.584192 19219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.589522 19219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40089
I20260927 02:51:30.589578 19346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40089 every 8 connection(s)
I20260927 02:51:30.589869 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:51:30.590632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 19219
I20260927 02:51:30.590742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:51:30.590952 392 external_mini_cluster.cc:1428] Setting key 2bdb73edc384f0c69a996b6caab98a0b
I20260927 02:51:30.591841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34505
--builtin_ntp_servers=127.0.98.20:45901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:51:30.595060 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.590520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52191 (local address 127.0.98.62:34505)
0927 02:51:30.590586 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.590588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.590826 (+ 238us) server_negotiation.cc:407] Connection header received
0927 02:51:30.591383 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.591385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.591394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.591462 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:30.592189 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.592682 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.593643 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.593899 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.593961 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.594765 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:30.594766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:30.594767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:30.594772 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:30.594788 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:30.594793 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:30.594866 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.594940 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.594969 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:51:30.595216 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.590511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34505 (local address 127.0.98.2:52191)
0927 02:51:30.590682 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.590723 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.591299 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.591530 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.591534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.591545 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:30.592104 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.592110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.592766 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.592767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.593560 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.593562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.593975 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.593976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.594100 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.594618 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:30.594618 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:30.594647 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:30.594673 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:30.594870 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:30.594873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:30.594876 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.594927 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.595025 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260927 02:51:30.595707 19347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34505
I20260927 02:51:30.595793 19347 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:30.595963 19347 heartbeater.cc:511] Master 127.0.98.62:34505 requested a full tablet report, sending...
I20260927 02:51:30.596300 19028 ts_manager.cc:194] Registered new tserver with Master: d1d2b130fbea4717bce1d26805017961 (127.0.98.2:40089)
I20260927 02:51:30.596659 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52191
W20260927 02:51:30.670395 19350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:30.670617 19350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:30.670645 19350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:30.671787 19350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:30.671826 19350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:30.671859 19350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:51:30.673228 19350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:30.673457 19350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:30.673727 19350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:30.678354 19356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:30.678478 19350 server_base.cc:1034] running on GCE node
W20260927 02:51:30.678359 19358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:30.678372 19355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:30.678751 19350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:30.679000 19350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:30.680143 19350 hybrid_clock.cc:648] HybridClock initialized: now 1790477490680124 us; error 36 us; skew 500 ppm
I20260927 02:51:30.681345 19350 webserver.cc:460] Webserver started at http://127.0.98.3:42273/ using document root <none> and password file <none>
I20260927 02:51:30.681566 19350 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:30.681648 19350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:30.681769 19350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:30.682878 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "4c44a33a53a3587880804886610cd50e"
server_key_iv: "a26e4b8ed988cecf50518bea2a068b9d"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.683219 19350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "4c44a33a53a3587880804886610cd50e"
server_key_iv: "a26e4b8ed988cecf50518bea2a068b9d"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.684456 19350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:30.685140 19364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.685287 19350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:30.685367 19350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca"
format_stamp: "Formatted at 2026-09-27 02:51:30 on dist-test-slave-f4rg"
server_key: "4c44a33a53a3587880804886610cd50e"
server_key_iv: "a26e4b8ed988cecf50518bea2a068b9d"
server_key_version: "encryptionkey@0"
I20260927 02:51:30.685431 19350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:30.696538 19350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:30.696828 19350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:30.696949 19350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:30.697152 19350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:30.697470 19350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:30.697503 19350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.697531 19350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:30.697549 19350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:30.702945 19350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42587
I20260927 02:51:30.702982 19477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42587 every 8 connection(s)
I20260927 02:51:30.703294 19350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:51:30.706264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 19350
I20260927 02:51:30.706357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:51:30.706519 392 external_mini_cluster.cc:1428] Setting key 666e891079897252aaaa62ac4b26ff24
I20260927 02:51:30.708352 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.703934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48219 (local address 127.0.98.62:34505)
0927 02:51:30.703985 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.703987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.704241 (+ 254us) server_negotiation.cc:407] Connection header received
0927 02:51:30.704712 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.704713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.704723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.704772 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:30.705403 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.705888 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.707039 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.707277 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.707342 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.708135 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:30.708136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:30.708136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:30.708140 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:30.708154 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:30.708158 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:30.708206 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.708269 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.708293 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 02:51:30.708544 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.703948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34505 (local address 127.0.98.3:48219)
0927 02:51:30.704118 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.704151 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.704623 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.704836 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.704839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.704849 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:30.705337 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.705342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.705993 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.705994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.706953 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.706956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.707355 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.707356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.707535 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.708015 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:30.708016 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:30.708043 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:30.708065 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:30.708210 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:30.708212 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:30.708214 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.708265 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.708343 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260927 02:51:30.708921 19478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34505
I20260927 02:51:30.708997 19478 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:30.709168 19478 heartbeater.cc:511] Master 127.0.98.62:34505 requested a full tablet report, sending...
I20260927 02:51:30.709493 19028 ts_manager.cc:194] Registered new tserver with Master: e33dcb4ea6bb4cc6a64faaa5ad6409ca (127.0.98.3:42587)
I20260927 02:51:30.709837 19028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48219
I20260927 02:51:30.718185 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:51:30.722204 19086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.719454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.0.98.62:34505)
0927 02:51:30.719499 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.719501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.719620 (+ 119us) server_negotiation.cc:407] Connection header received
0927 02:51:30.719683 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.719684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.719696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.719738 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:30.719886 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.720362 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.721100 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.721278 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.721330 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.721944 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:30.721945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:30.721945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:30.721949 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:30.721963 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:30.721967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:30.722037 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.722112 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.722122 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:51:30.723644 19028 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:51:30.725143 19028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:51:30.731272 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.727010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40089 (local address 127.0.0.1:41388)
0927 02:51:30.727291 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.727330 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.727397 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.727712 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.727715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.727718 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.727785 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.727787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.729210 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.729211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.730296 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.730298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.730993 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.730994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.731054 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.731073 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.731131 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.731190 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260927 02:51:30.731294 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.727103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.0.98.1:45053)
0927 02:51:30.727276 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.727282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.727424 (+ 142us) server_negotiation.cc:407] Connection header received
0927 02:51:30.727557 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.727559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.727574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.727625 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.727885 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.729049 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.730110 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.730741 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.730786 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.730902 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.731172 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.731211 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260927 02:51:30.731357 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.727235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42587 (local address 127.0.0.1:59262)
0927 02:51:30.727512 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.727520 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.727554 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.727851 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.727852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.727854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.727905 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.727906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.728970 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.728971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.730262 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.730265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.731137 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.731137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.731189 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.731192 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.731228 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.731257 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":79,"threads_started":1}
I20260927 02:51:30.731505 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.727010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45053 (local address 127.0.0.1:53374)
0927 02:51:30.727366 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:30.727373 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:30.727462 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:30.727715 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:30.727716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:30.727721 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:30.727797 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.727799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.729140 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.729141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.730024 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:30.730026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.730975 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.730976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:30.731054 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.731068 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 02:51:30.731395 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 02:51:30.731440 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":47,"thread_start_us":85,"threads_started":1}
I20260927 02:51:30.731565 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.727439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.0.98.3:42587)
0927 02:51:30.727707 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.727712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.727721 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:30.727756 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.727758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.727771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.727846 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.728010 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.728895 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.730344 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.731073 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.731138 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.731218 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.731242 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.731275 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":147,"threads_started":1}
I20260927 02:51:30.731884 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:30.727102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.98.2:40089)
0927 02:51:30.727370 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:30.727374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:30.727385 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:51:30.727514 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:30.727516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:30.727533 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:30.727608 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:30.727999 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.729133 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.730375 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:30.730921 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:30.730982 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:30.731068 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:30.731762 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 02:51:30.731806 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260927 02:51:30.732178 19412 tablet_service.cc:1467] Processing CreateTablet for tablet 7886a9e146ac44af8a4825825b9a6547 (DEFAULT_TABLE table=test-table [id=861fcd30c4164c1f97edc53005b1eed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:30.732537 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7886a9e146ac44af8a4825825b9a6547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:30.732498 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 7886a9e146ac44af8a4825825b9a6547 (DEFAULT_TABLE table=test-table [id=861fcd30c4164c1f97edc53005b1eed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:30.732777 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7886a9e146ac44af8a4825825b9a6547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:30.734967 19492 tablet_bootstrap.cc:492] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Bootstrap starting.
I20260927 02:51:30.735260 19150 tablet_service.cc:1467] Processing CreateTablet for tablet 7886a9e146ac44af8a4825825b9a6547 (DEFAULT_TABLE table=test-table [id=861fcd30c4164c1f97edc53005b1eed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:30.735342 19493 tablet_bootstrap.cc:492] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Bootstrap starting.
I20260927 02:51:30.735533 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7886a9e146ac44af8a4825825b9a6547. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:30.735620 19492 tablet_bootstrap.cc:654] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:30.735909 19493 tablet_bootstrap.cc:654] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:30.735916 19492 log.cc:824] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:30.736191 19493 log.cc:824] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:30.736658 19492 tablet_bootstrap.cc:492] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: No bootstrap required, opened a new log
I20260927 02:51:30.736706 19492 ts_tablet_manager.cc:1397] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:30.736863 19493 tablet_bootstrap.cc:492] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: No bootstrap required, opened a new log
I20260927 02:51:30.736922 19493 ts_tablet_manager.cc:1397] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:30.738395 19496 tablet_bootstrap.cc:492] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538: Bootstrap starting.
I20260927 02:51:30.738395 19492 raft_consensus.cc:348] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:30.738544 19492 raft_consensus.cc:374] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:30.738576 19492 raft_consensus.cc:729] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33dcb4ea6bb4cc6a64faaa5ad6409ca, State: Initialized, Role: FOLLOWER
I20260927 02:51:30.738727 19492 consensus_queue.cc:260] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:30.738998 19492 ts_tablet_manager.cc:1428] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:51:30.739028 19496 tablet_bootstrap.cc:654] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:30.739121 19478 heartbeater.cc:503] Master 127.0.98.62:34505 was elected leader, sending a full tablet report...
I20260927 02:51:30.739308 19496 log.cc:824] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:30.739600 19493 raft_consensus.cc:348] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:30.739734 19493 raft_consensus.cc:374] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:30.739768 19493 raft_consensus.cc:729] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1d2b130fbea4717bce1d26805017961, State: Initialized, Role: FOLLOWER
I20260927 02:51:30.739892 19493 consensus_queue.cc:260] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:30.739993 19496 tablet_bootstrap.cc:492] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538: No bootstrap required, opened a new log
I20260927 02:51:30.740048 19496 ts_tablet_manager.cc:1397] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:30.740140 19493 ts_tablet_manager.cc:1428] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:30.740244 19347 heartbeater.cc:503] Master 127.0.98.62:34505 was elected leader, sending a full tablet report...
I20260927 02:51:30.741711 19496 raft_consensus.cc:348] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:30.741820 19496 raft_consensus.cc:374] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:30.741838 19496 raft_consensus.cc:729] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d548f6a5d94234bc30a7c6d3d12538, State: Initialized, Role: FOLLOWER
I20260927 02:51:30.741914 19496 consensus_queue.cc:260] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:30.742080 19496 ts_tablet_manager.cc:1428] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:51:30.742152 19216 heartbeater.cc:503] Master 127.0.98.62:34505 was elected leader, sending a full tablet report...
W20260927 02:51:30.840410 19348 tablet.cc:2369] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:51:30.953800 19479 tablet.cc:2369] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 02:51:30.970546 19217 tablet.cc:2369] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:51:31.012743 19499 raft_consensus.cc:482] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:51:31.012861 19499 raft_consensus.cc:504] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:31.013237 19499 leader_election.cc:288] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29d548f6a5d94234bc30a7c6d3d12538 (127.0.98.1:45053), e33dcb4ea6bb4cc6a64faaa5ad6409ca (127.0.98.3:42587)
I20260927 02:51:31.015888 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:31.013481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57383 (local address 127.0.98.1:45053)
0927 02:51:31.013528 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:31.013530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:31.013541 (+ 11us) server_negotiation.cc:407] Connection header received
0927 02:51:31.013647 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:31.013648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:31.013658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:31.013704 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:31.013841 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.014317 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.015256 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.015603 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.015661 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:31.015670 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:31.015787 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:51:31.015822 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 02:51:31.015887 19349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:31.013395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45053 (local address 127.0.98.2:57383)
0927 02:51:31.013457 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:31.013471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:31.013534 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:31.013717 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:31.013718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:31.013719 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:31.013784 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:31.013785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.014392 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.014393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:31.015177 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:31.015180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.015669 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.015670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:31.015730 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:31.015733 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:31.015776 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:51:31.015817 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:51:31.015995 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:31.013489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42587 (local address 127.0.98.2:41865)
0927 02:51:31.013644 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:31.013652 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:31.013711 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:31.013832 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:31.013833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:31.013834 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:31.013905 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:31.013906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.014464 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.014464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:31.015254 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:31.015256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.015749 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.015750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:31.015830 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:31.015833 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:31.015876 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:51:31.015911 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260927 02:51:31.015983 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:31.013564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41865 (local address 127.0.98.3:42587)
0927 02:51:31.013608 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:31.013610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:31.013708 (+ 98us) server_negotiation.cc:407] Connection header received
0927 02:51:31.013771 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:31.013772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:31.013780 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:31.013826 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:31.013956 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.014399 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.015329 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.015682 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.015738 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:31.015747 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:31.015886 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:51:31.015920 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:51:31.016274 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886a9e146ac44af8a4825825b9a6547" candidate_uuid: "d1d2b130fbea4717bce1d26805017961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" is_pre_election: true
I20260927 02:51:31.016273 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886a9e146ac44af8a4825825b9a6547" candidate_uuid: "d1d2b130fbea4717bce1d26805017961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d548f6a5d94234bc30a7c6d3d12538" is_pre_election: true
I20260927 02:51:31.016419 19170 raft_consensus.cc:2454] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d2b130fbea4717bce1d26805017961 in term 0.
I20260927 02:51:31.016419 19432 raft_consensus.cc:2454] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1d2b130fbea4717bce1d26805017961 in term 0.
I20260927 02:51:31.016645 19237 leader_election.cc:302] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1d2b130fbea4717bce1d26805017961, e33dcb4ea6bb4cc6a64faaa5ad6409ca; no voters:
I20260927 02:51:31.016769 19499 raft_consensus.cc:2781] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:51:31.016839 19499 raft_consensus.cc:482] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:51:31.016857 19499 raft_consensus.cc:3037] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:31.017532 19499 raft_consensus.cc:504] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:31.017630 19499 leader_election.cc:288] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [CANDIDATE]: Term 1 election: Requested vote from peers 29d548f6a5d94234bc30a7c6d3d12538 (127.0.98.1:45053), e33dcb4ea6bb4cc6a64faaa5ad6409ca (127.0.98.3:42587)
I20260927 02:51:31.017800 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886a9e146ac44af8a4825825b9a6547" candidate_uuid: "d1d2b130fbea4717bce1d26805017961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29d548f6a5d94234bc30a7c6d3d12538"
I20260927 02:51:31.017864 19170 raft_consensus.cc:3037] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:31.017825 19432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7886a9e146ac44af8a4825825b9a6547" candidate_uuid: "d1d2b130fbea4717bce1d26805017961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca"
I20260927 02:51:31.017926 19432 raft_consensus.cc:3037] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:31.018576 19170 raft_consensus.cc:2454] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d2b130fbea4717bce1d26805017961 in term 1.
I20260927 02:51:31.018622 19432 raft_consensus.cc:2454] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1d2b130fbea4717bce1d26805017961 in term 1.
I20260927 02:51:31.018743 19237 leader_election.cc:302] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1d2b130fbea4717bce1d26805017961, e33dcb4ea6bb4cc6a64faaa5ad6409ca; no voters:
I20260927 02:51:31.018872 19499 raft_consensus.cc:2781] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:31.019006 19499 raft_consensus.cc:686] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 1 LEADER]: Becoming Leader. State: Replica: d1d2b130fbea4717bce1d26805017961, State: Running, Role: LEADER
I20260927 02:51:31.019093 19499 consensus_queue.cc:237] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } }
I20260927 02:51:31.019894 19027 catalog_manager.cc:5495] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1d2b130fbea4717bce1d26805017961 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d1d2b130fbea4717bce1d26805017961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1d2b130fbea4717bce1d26805017961" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20260927 02:51:31.082221 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:31.079956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.98.2:40089)
0927 02:51:31.080017 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:31.080019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:31.080026 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:51:31.080060 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:31.080061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:31.080082 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:31.080141 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:31.080265 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.080731 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.081446 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:31.081641 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:31.081695 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:31.081876 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:31.081978 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:31.082066 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:31.082113 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:51:31.082124 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:51:31.083941 19170 raft_consensus.cc:1261] T 7886a9e146ac44af8a4825825b9a6547 P 29d548f6a5d94234bc30a7c6d3d12538 [term 1 FOLLOWER]: Refusing update from remote peer d1d2b130fbea4717bce1d26805017961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:31.083935 19432 raft_consensus.cc:1261] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 1 FOLLOWER]: Refusing update from remote peer d1d2b130fbea4717bce1d26805017961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:31.084261 19499 consensus_queue.cc:1035] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29d548f6a5d94234bc30a7c6d3d12538" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:31.084508 19499 consensus_queue.cc:1035] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33dcb4ea6bb4cc6a64faaa5ad6409ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:31.086280 19505 mvcc.cc:205] Tried to move back new op lower bound from 7333795803477291008 to 7333795803214708736. Current Snapshot: MvccSnapshot[applied={T|T < 7333795803477291008}]
I20260927 02:51:31.088979 19028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:31.089079 19028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:31.089758 19028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abfff5e4e65648b685f1c06f0bac56e4: Sending DeleteTablet for 3 replicas of tablet 7886a9e146ac44af8a4825825b9a6547
I20260927 02:51:31.090080 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 7886a9e146ac44af8a4825825b9a6547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:31 UTC) from {username='slave'} at 127.0.0.1:41388
I20260927 02:51:31.090097 19150 tablet_service.cc:1514] Processing DeleteTablet for tablet 7886a9e146ac44af8a4825825b9a6547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:31 UTC) from {username='slave'} at 127.0.0.1:53374
I20260927 02:51:31.090162 19412 tablet_service.cc:1514] Processing DeleteTablet for tablet 7886a9e146ac44af8a4825825b9a6547 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:31 UTC) from {username='slave'} at 127.0.0.1:59262
I20260927 02:51:31.090269 19514 tablet_replica.cc:330] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: stopping tablet replica
I20260927 02:51:31.090350 19514 raft_consensus.cc:2229] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:51:31.090440 19514 raft_consensus.cc:2258] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:31.090535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 19088
I20260927 02:51:31.091574 19514 ts_tablet_manager.cc:1905] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:31.093159 19514 ts_tablet_manager.cc:1918] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:31.093233 19514 log.cc:1196] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/7886a9e146ac44af8a4825825b9a6547
I20260927 02:51:31.093473 19514 ts_tablet_manager.cc:1939] T 7886a9e146ac44af8a4825825b9a6547 P d1d2b130fbea4717bce1d26805017961: Deleting consensus metadata
I20260927 02:51:31.093851 19013 catalog_manager.cc:4841] TS d1d2b130fbea4717bce1d26805017961 (127.0.98.2:40089): tablet 7886a9e146ac44af8a4825825b9a6547 (table test-table [id=861fcd30c4164c1f97edc53005b1eed6]) successfully deleted
W20260927 02:51:31.097376 19015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45053 (error 108)
I20260927 02:51:31.097579 19516 tablet_replica.cc:330] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: stopping tablet replica
I20260927 02:51:31.097780 19516 raft_consensus.cc:2229] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:31.097821 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:31.097697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45053 (local address 127.0.0.1:53376)
0927 02:51:31.097752 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:31.097790 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:51:31.097909 19516 raft_consensus.cc:2258] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 02:51:31.097921 19015 catalog_manager.cc:4584] TS 29d548f6a5d94234bc30a7c6d3d12538 (127.0.98.1:45053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7886a9e146ac44af8a4825825b9a6547: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45053: connect: Connection refused (error 111)
I20260927 02:51:31.098093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 19219
I20260927 02:51:31.099668 19516 ts_tablet_manager.cc:1905] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:31.101270 19516 ts_tablet_manager.cc:1918] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:31.101353 19516 log.cc:1196] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/7886a9e146ac44af8a4825825b9a6547
I20260927 02:51:31.101599 19516 ts_tablet_manager.cc:1939] T 7886a9e146ac44af8a4825825b9a6547 P e33dcb4ea6bb4cc6a64faaa5ad6409ca: Deleting consensus metadata
I20260927 02:51:31.102007 19015 catalog_manager.cc:4841] TS e33dcb4ea6bb4cc6a64faaa5ad6409ca (127.0.98.3:42587): tablet 7886a9e146ac44af8a4825825b9a6547 (table test-table [id=861fcd30c4164c1f97edc53005b1eed6]) successfully deleted
I20260927 02:51:31.102973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 19350
I20260927 02:51:31.107327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 18997
2026-09-27T02:51:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (925 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:31 dist-test-slave-f4rg krb5kdc[19520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:31 dist-test-slave-f4rg krb5kdc[19520](info): set up 2 sockets
Sep 27 02:51:31 dist-test-slave-f4rg krb5kdc[19520](info): commencing operation
krb5kdc: starting...
W20260927 02:51:32.054400 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:32.165621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39627
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:32.244879 19533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:32.245106 19533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:32.245129 19533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:32.246323 19533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:32.246371 19533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:32.246384 19533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:32.246395 19533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:32.246407 19533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:32.247980 19533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:32.248217 19533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:32.248436 19533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:32.253264 19538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:32.253284 19541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.253327 19533 server_base.cc:1034] running on GCE node
W20260927 02:51:32.253242 19539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.253667 19533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:32.253940 19533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:32.255072 19533 hybrid_clock.cc:648] HybridClock initialized: now 1790477492255059 us; error 38 us; skew 500 ppm
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:32.257445 19533 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:32.257864 19533 webserver.cc:460] Webserver started at http://127.0.98.62:33747/ using document root <none> and password file <none>
I20260927 02:51:32.258051 19533 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:32.258090 19533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:32.258184 19533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:32.259218 19533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "d558c213b3f6427cacc972888db3c300"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "5d8984d7e1217c1201e58d3fcc950e70"
server_key_iv: "b25e9c0076df6eb9c290bb25368a7eea"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.259505 19533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "d558c213b3f6427cacc972888db3c300"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "5d8984d7e1217c1201e58d3fcc950e70"
server_key_iv: "b25e9c0076df6eb9c290bb25368a7eea"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.260695 19533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:32.261453 19548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.261669 19533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:32.261754 19533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "d558c213b3f6427cacc972888db3c300"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "5d8984d7e1217c1201e58d3fcc950e70"
server_key_iv: "b25e9c0076df6eb9c290bb25368a7eea"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.261811 19533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:32.269600 19533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:32.271335 19533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:32.271440 19533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:32.275123 19533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39627
I20260927 02:51:32.275159 19600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39627 every 8 connection(s)
I20260927 02:51:32.275468 19533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:32.276261 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:32.278388 19601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300: Bootstrap starting.
I20260927 02:51:32.278934 19601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:32.279155 19601 log.cc:824] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:32.279762 19601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300: No bootstrap required, opened a new log
I20260927 02:51:32.280205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 19533
I20260927 02:51:32.280314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:32.280505 392 external_mini_cluster.cc:1428] Setting key 77a3aefdcb0b56382bcfa715e6bf245a
I20260927 02:51:32.281785 19601 raft_consensus.cc:348] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d558c213b3f6427cacc972888db3c300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39627 } }
I20260927 02:51:32.281922 19601 raft_consensus.cc:374] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:32.281952 19601 raft_consensus.cc:729] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558c213b3f6427cacc972888db3c300, State: Initialized, Role: FOLLOWER
I20260927 02:51:32.282071 19601 consensus_queue.cc:260] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d558c213b3f6427cacc972888db3c300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39627 } }
I20260927 02:51:32.282147 19601 raft_consensus.cc:388] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:32.282176 19601 raft_consensus.cc:482] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:32.282243 19601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:32.282950 19601 raft_consensus.cc:504] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d558c213b3f6427cacc972888db3c300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39627 } }
I20260927 02:51:32.283038 19601 leader_election.cc:302] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d558c213b3f6427cacc972888db3c300; no voters:
I20260927 02:51:32.283188 19601 leader_election.cc:288] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:32.283233 19606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:32.283387 19606 raft_consensus.cc:686] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [term 1 LEADER]: Becoming Leader. State: Replica: d558c213b3f6427cacc972888db3c300, State: Running, Role: LEADER
I20260927 02:51:32.283433 19601 sys_catalog.cc:564] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:32.283547 19606 consensus_queue.cc:237] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d558c213b3f6427cacc972888db3c300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39627 } }
I20260927 02:51:32.283883 19606 sys_catalog.cc:455] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d558c213b3f6427cacc972888db3c300. Latest consensus state: current_term: 1 leader_uuid: "d558c213b3f6427cacc972888db3c300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d558c213b3f6427cacc972888db3c300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39627 } } }
I20260927 02:51:32.283946 19606 sys_catalog.cc:458] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:32.284077 19607 sys_catalog.cc:455] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d558c213b3f6427cacc972888db3c300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d558c213b3f6427cacc972888db3c300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39627 } } }
I20260927 02:51:32.284123 19607 sys_catalog.cc:458] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:32.284209 19614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:32.284686 19614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:32.286958 19614 catalog_manager.cc:1347] Generated new cluster ID: 17bbecd22e174eeb8835cf1e7611aa3b
I20260927 02:51:32.287005 19614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:32.291896 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.281005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.0.98.62:39627)
0927 02:51:32.281167 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.281174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.281198 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:51:32.281949 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.281957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.282032 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.282169 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:32.282899 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.284244 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.285292 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.285678 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.285749 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.288381 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:32.288385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:32.288386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:32.288399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:32.290283 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.290642 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.290645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.290645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.290764 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.290954 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.290955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.290955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.291257 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:32.291425 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.291522 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.291664 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260927 02:51:32.299108 19614 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:32.299609 19614 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:32.302284 19614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300: Generated new TSK 0
I20260927 02:51:32.302435 19614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 02:51:32.331208 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39627
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:32.409978 19627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:32.410229 19627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:32.410254 19627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:32.411473 19627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:32.411520 19627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:32.411545 19627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:51:32.413054 19627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:32.413304 19627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:32.413519 19627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:32.418249 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:32.418253 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:32.418275 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.418392 19627 server_base.cc:1034] running on GCE node
I20260927 02:51:32.418692 19627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:32.418933 19627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:32.420068 19627 hybrid_clock.cc:648] HybridClock initialized: now 1790477492420051 us; error 32 us; skew 500 ppm
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:32.422430 19627 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 02:51:32.422891 19627 webserver.cc:460] Webserver started at http://127.0.98.1:40259/ using document root <none> and password file <none>
I20260927 02:51:32.423101 19627 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:32.423144 19627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:32.423264 19627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:32.424260 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "702a62fb1b904d5e8fc2d949a0b967a6"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "79b25c853da12ea658c676aa05521cb2"
server_key_iv: "4906e1ae45b34318155ee56c74e76c92"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.424588 19627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "702a62fb1b904d5e8fc2d949a0b967a6"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "79b25c853da12ea658c676aa05521cb2"
server_key_iv: "4906e1ae45b34318155ee56c74e76c92"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.425815 19627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:51:32.426533 19642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.426685 19627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:32.426757 19627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "702a62fb1b904d5e8fc2d949a0b967a6"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "79b25c853da12ea658c676aa05521cb2"
server_key_iv: "4906e1ae45b34318155ee56c74e76c92"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.426846 19627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:32.435647 19627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:32.437479 19627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:32.437588 19627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:32.437808 19627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:32.438097 19627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:32.438130 19627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.438158 19627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:32.438174 19627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.443514 19627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46297
I20260927 02:51:32.443554 19755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46297 every 8 connection(s)
I20260927 02:51:32.443856 19627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:51:32.445595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 19627
I20260927 02:51:32.445716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:51:32.445941 392 external_mini_cluster.cc:1428] Setting key 539876af178b048c72ec5c802f783698
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 02:51:32.453836 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.444568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47991 (local address 127.0.98.62:39627)
0927 02:51:32.444623 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.444624 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.444811 (+ 187us) server_negotiation.cc:407] Connection header received
0927 02:51:32.445738 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.445739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.445751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.445804 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:32.446587 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.447196 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.448445 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.448754 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.448821 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.451220 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:32.451221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:32.451221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:32.451228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:32.452477 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.452902 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.452903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.452903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.452949 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.453224 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.453224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.453225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.453316 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:32.453360 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.453678 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.453769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 02:51:32.454041 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.444533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39627 (local address 127.0.98.1:47991)
0927 02:51:32.444692 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.444733 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.445633 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.445908 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.445913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.446270 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:32.446489 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.446495 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.447278 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.447280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.448355 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.448358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.448834 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.448835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.448990 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.449489 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:32.449499 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:32.451120 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:32.452554 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:32.452560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:32.452562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:32.452833 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:32.452999 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:32.452999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:32.453000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:32.453132 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:32.453371 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:32.453378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:32.453538 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.453652 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.453819 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260927 02:51:32.454558 19756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39627
I20260927 02:51:32.454666 19756 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:32.454869 19756 heartbeater.cc:511] Master 127.0.98.62:39627 requested a full tablet report, sending...
I20260927 02:51:32.455343 19565 ts_manager.cc:194] Registered new tserver with Master: 702a62fb1b904d5e8fc2d949a0b967a6 (127.0.98.1:46297)
I20260927 02:51:32.455950 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 02:51:32.469487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39627
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:32.548563 19761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:32.548763 19761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:32.548785 19761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:32.549890 19761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:32.549924 19761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:32.549948 19761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:51:32.551492 19761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:32.551687 19761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:32.551854 19761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:32.556560 19767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:32.556608 19769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.556668 19761 server_base.cc:1034] running on GCE node
W20260927 02:51:32.556654 19766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.557008 19761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:32.557248 19761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:32.558394 19761 hybrid_clock.cc:648] HybridClock initialized: now 1790477492558383 us; error 28 us; skew 500 ppm
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:32.560737 19761 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 02:51:32.561134 19761 webserver.cc:460] Webserver started at http://127.0.98.2:32873/ using document root <none> and password file <none>
I20260927 02:51:32.561336 19761 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:32.561383 19761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:32.561503 19761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:32.562517 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "20b8b850b81f434193cc0ce9234b1db9"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "287b203ac87bcd1477f46d320954cd8d"
server_key_iv: "4c4e2e140daade32ff86324be9ed826f"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.562892 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20b8b850b81f434193cc0ce9234b1db9"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "287b203ac87bcd1477f46d320954cd8d"
server_key_iv: "4c4e2e140daade32ff86324be9ed826f"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.564237 19761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:32.564890 19776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.565035 19761 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.565112 19761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "20b8b850b81f434193cc0ce9234b1db9"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "287b203ac87bcd1477f46d320954cd8d"
server_key_iv: "4c4e2e140daade32ff86324be9ed826f"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.565178 19761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:32.574258 19761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:32.576032 19761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:32.576164 19761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:32.576363 19761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:32.576637 19761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:32.576668 19761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.576696 19761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:32.576713 19761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.581939 19761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41235
I20260927 02:51:32.581998 19889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41235 every 8 connection(s)
I20260927 02:51:32.582278 19761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:51:32.583869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 19761
I20260927 02:51:32.584025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:51:32.584234 392 external_mini_cluster.cc:1428] Setting key 02510a10e251e73e5dde4718237ee7a7
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:32.592052 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.582919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41749 (local address 127.0.98.62:39627)
0927 02:51:32.582979 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.582981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.583170 (+ 189us) server_negotiation.cc:407] Connection header received
0927 02:51:32.584295 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.584297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.584310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.584367 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:32.585138 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.585619 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.586763 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.587052 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.587120 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.589363 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:32.589365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:32.589365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:32.589374 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:32.590851 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.591197 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.591198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.591199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.591253 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.591447 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.591448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.591449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.591580 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:32.591659 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.591880 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.591955 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:51:32.592118 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.582906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39627 (local address 127.0.98.2:41749)
0927 02:51:32.583057 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.583093 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.584190 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.584443 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.584447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.584754 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:32.585029 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.585036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.585702 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.585703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.586676 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.586679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.587123 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.587123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.587243 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.587759 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:32.587767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:32.589275 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:32.590920 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:32.590925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:32.590927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:32.591130 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:32.591309 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:32.591310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:32.591310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:32.591395 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:32.591647 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:32.591649 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:32.591764 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.591839 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.591955 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 02:51:32.592540 19890 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39627
I20260927 02:51:32.592626 19890 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:32.592769 19890 heartbeater.cc:511] Master 127.0.98.62:39627 requested a full tablet report, sending...
I20260927 02:51:32.593075 19565 ts_manager.cc:194] Registered new tserver with Master: 20b8b850b81f434193cc0ce9234b1db9 (127.0.98.2:41235)
I20260927 02:51:32.593449 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 02:51:32.609920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39627
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:32.688854 19895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:32.689075 19895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:32.689100 19895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:32.690253 19895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:32.690291 19895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:32.690316 19895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:51:32.691782 19895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:32.691975 19895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:32.692160 19895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:32.696879 19900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.696957 19895 server_base.cc:1034] running on GCE node
W20260927 02:51:32.696882 19903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:32.697054 19901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.697299 19895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:32.697570 19895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:32.698751 19895 hybrid_clock.cc:648] HybridClock initialized: now 1790477492698734 us; error 33 us; skew 500 ppm
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:32.701166 19895 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 02:51:32.701640 19895 webserver.cc:460] Webserver started at http://127.0.98.3:32991/ using document root <none> and password file <none>
I20260927 02:51:32.701851 19895 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:32.701897 19895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:32.702028 19895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:32.703194 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "4e7b6e28f60145f5aea544ffb7c615ff"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "daca3e746b8cf8111da26b2af40358d2"
server_key_iv: "6f1ab8e8832bed03cc552838cc9d449d"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.703635 19895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4e7b6e28f60145f5aea544ffb7c615ff"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "daca3e746b8cf8111da26b2af40358d2"
server_key_iv: "6f1ab8e8832bed03cc552838cc9d449d"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.704867 19895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:32.705540 19910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.705686 19895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:32.705767 19895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "4e7b6e28f60145f5aea544ffb7c615ff"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "daca3e746b8cf8111da26b2af40358d2"
server_key_iv: "6f1ab8e8832bed03cc552838cc9d449d"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.705825 19895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:32.724152 19895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:32.725951 19895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:32.726075 19895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:32.726300 19895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:32.726603 19895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:32.726636 19895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.726657 19895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:32.726679 19895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.732193 19895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36769
I20260927 02:51:32.732236 20023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36769 every 8 connection(s)
I20260927 02:51:32.732537 19895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:51:32.734467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 19895
I20260927 02:51:32.734581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:51:32.734791 392 external_mini_cluster.cc:1428] Setting key f0e0145e41a6d23b37884100de2972f8
Sep 27 02:51:32 dist-test-slave-f4rg krb5kdc[19520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:32.741285 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.733171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41117 (local address 127.0.98.62:39627)
0927 02:51:32.733245 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.733247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.733399 (+ 152us) server_negotiation.cc:407] Connection header received
0927 02:51:32.734289 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.734290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.734300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.734350 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:32.735209 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.735806 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.736858 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.737033 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.737077 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.738958 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:32.738959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:32.738959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:32.738966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:32.740187 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.740595 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.740595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.740596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.740639 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.740828 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.740829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.740829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.740923 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:32.740968 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.741169 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.741222 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 02:51:32.741533 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.733130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39627 (local address 127.0.98.3:41117)
0927 02:51:32.733279 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.733320 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.734194 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.734431 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.734436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.734795 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:32.735086 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.735092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.735876 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.735877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.736793 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.736796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.737078 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.737079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.737172 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.737577 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:32.737584 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:32.738873 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:32.740262 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:32.740269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:32.740271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:32.740539 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:32.740688 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:32.740688 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:32.740689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:32.740779 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:32.740967 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:32.740970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:32.741083 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.741158 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.741274 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260927 02:51:32.742017 20024 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39627
I20260927 02:51:32.742100 20024 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:32.742266 20024 heartbeater.cc:511] Master 127.0.98.62:39627 requested a full tablet report, sending...
I20260927 02:51:32.742642 19565 ts_manager.cc:194] Registered new tserver with Master: 4e7b6e28f60145f5aea544ffb7c615ff (127.0.98.3:36769)
I20260927 02:51:32.743057 19565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41117
I20260927 02:51:32.746479 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:51:32.753226 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.747780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.98.62:39627)
0927 02:51:32.747844 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.747845 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.747952 (+ 107us) server_negotiation.cc:407] Connection header received
0927 02:51:32.748015 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.748016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.748026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.748070 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:32.748588 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.749073 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.749774 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.749972 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.750023 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.751147 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:32.751149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:32.751149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:32.751156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:32.752173 (+ 1017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.752462 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.752463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.752464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.752518 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:32.752713 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:32.752714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:32.752714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:32.752856 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:32.752939 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.753065 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.753138 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 02:51:32.754494 19565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:51:32.755980 19565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:51:32.761868 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.757871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36769 (local address 127.0.0.1:51594)
0927 02:51:32.758217 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.758227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.758277 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.758522 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.758523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.758526 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:32.758590 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.758592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.759849 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.759850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.760888 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.760890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.761586 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.761587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.761665 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.761678 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.761739 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.761789 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260927 02:51:32.761950 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.757991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.0.98.2:41235)
0927 02:51:32.758204 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.758209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.758218 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:32.758260 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.758262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.758278 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.758350 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:32.758601 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.759542 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.760943 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.761662 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.761733 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.761787 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.761843 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.761878 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
I20260927 02:51:32.761988 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.757871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41235 (local address 127.0.0.1:34078)
0927 02:51:32.758017 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.758060 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.758125 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.758439 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.758442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.758444 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:32.758503 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.758504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.759629 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.759630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.760858 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.760861 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.761730 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.761731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.761782 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.761785 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.761832 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.761862 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"spinlock_wait_cycles":384,"thread_start_us":84,"threads_started":1}
I20260927 02:51:32.762187 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.758044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.0.98.3:36769)
0927 02:51:32.758397 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.758402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.758410 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:32.758440 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.758442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.758453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.758515 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:32.758661 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.759753 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.760969 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.761508 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.761578 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.761637 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.762068 (+ 431us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.762112 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":85,"threads_started":1}
I20260927 02:51:32.762496 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.758064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46297 (local address 127.0.0.1:54488)
0927 02:51:32.758276 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.758284 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.758319 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.758860 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.758861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.758863 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:32.758898 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.758900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.760044 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.760045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.760879 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.760881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.761893 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.761894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.762345 (+ 451us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.762350 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.762400 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.762438 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260927 02:51:32.762550 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.758284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.0.98.1:46297)
0927 02:51:32.758629 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.758633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.758643 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:51:32.758676 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.758678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.758691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.758757 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:32.758959 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.759965 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.760957 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.761662 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.761733 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.761787 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.762429 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.762470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":76,"threads_started":1}
I20260927 02:51:32.762672 19824 tablet_service.cc:1467] Processing CreateTablet for tablet 7afbbe0686d142a39d9f5570b4d5f7fe (DEFAULT_TABLE table=test-table [id=8c7926e3dd8e44b0bc67ad3c8a91d3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:32.763047 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afbbe0686d142a39d9f5570b4d5f7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:32.763157 19958 tablet_service.cc:1467] Processing CreateTablet for tablet 7afbbe0686d142a39d9f5570b4d5f7fe (DEFAULT_TABLE table=test-table [id=8c7926e3dd8e44b0bc67ad3c8a91d3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:32.763191 19690 tablet_service.cc:1467] Processing CreateTablet for tablet 7afbbe0686d142a39d9f5570b4d5f7fe (DEFAULT_TABLE table=test-table [id=8c7926e3dd8e44b0bc67ad3c8a91d3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:32.763389 19958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afbbe0686d142a39d9f5570b4d5f7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:32.763473 19690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afbbe0686d142a39d9f5570b4d5f7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:32.765771 20038 tablet_bootstrap.cc:492] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Bootstrap starting.
I20260927 02:51:32.766202 20039 tablet_bootstrap.cc:492] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Bootstrap starting.
I20260927 02:51:32.766342 20038 tablet_bootstrap.cc:654] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:32.766851 20039 tablet_bootstrap.cc:654] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:32.766891 20040 tablet_bootstrap.cc:492] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6: Bootstrap starting.
I20260927 02:51:32.766920 20038 log.cc:824] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:32.767334 20039 log.cc:824] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:32.767468 20040 tablet_bootstrap.cc:654] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:32.767735 20038 tablet_bootstrap.cc:492] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: No bootstrap required, opened a new log
I20260927 02:51:32.767808 20038 ts_tablet_manager.cc:1397] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:32.768015 20039 tablet_bootstrap.cc:492] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: No bootstrap required, opened a new log
I20260927 02:51:32.768005 20040 log.cc:824] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:32.768080 20039 ts_tablet_manager.cc:1397] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:32.768731 20040 tablet_bootstrap.cc:492] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6: No bootstrap required, opened a new log
I20260927 02:51:32.768793 20040 ts_tablet_manager.cc:1397] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:32.769661 20039 raft_consensus.cc:348] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.769821 20039 raft_consensus.cc:374] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:32.769855 20039 raft_consensus.cc:729] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b8b850b81f434193cc0ce9234b1db9, State: Initialized, Role: FOLLOWER
I20260927 02:51:32.769837 20038 raft_consensus.cc:348] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.769963 20039 consensus_queue.cc:260] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.769999 20038 raft_consensus.cc:374] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:32.770035 20038 raft_consensus.cc:729] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7b6e28f60145f5aea544ffb7c615ff, State: Initialized, Role: FOLLOWER
I20260927 02:51:32.770208 20039 ts_tablet_manager.cc:1428] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 02:51:32.770164 20038 consensus_queue.cc:260] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.770328 19890 heartbeater.cc:503] Master 127.0.98.62:39627 was elected leader, sending a full tablet report...
I20260927 02:51:32.770433 20038 ts_tablet_manager.cc:1428] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:32.770479 20024 heartbeater.cc:503] Master 127.0.98.62:39627 was elected leader, sending a full tablet report...
I20260927 02:51:32.770952 20040 raft_consensus.cc:348] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.771232 20040 raft_consensus.cc:374] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:32.771272 20040 raft_consensus.cc:729] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702a62fb1b904d5e8fc2d949a0b967a6, State: Initialized, Role: FOLLOWER
I20260927 02:51:32.771377 20040 consensus_queue.cc:260] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.771612 19756 heartbeater.cc:503] Master 127.0.98.62:39627 was elected leader, sending a full tablet report...
I20260927 02:51:32.771615 20040 ts_tablet_manager.cc:1428] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:32.832662 20045 raft_consensus.cc:482] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 02:51:32.832772 19891 tablet.cc:2369] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:51:32.832789 20045 raft_consensus.cc:504] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.833142 20045 leader_election.cc:288] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b8b850b81f434193cc0ce9234b1db9 (127.0.98.2:41235), 702a62fb1b904d5e8fc2d949a0b967a6 (127.0.98.1:46297)
I20260927 02:51:32.836217 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.833451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36849 (local address 127.0.98.1:46297)
0927 02:51:32.833507 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.833509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.833708 (+ 199us) server_negotiation.cc:407] Connection header received
0927 02:51:32.833741 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.833742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.833753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.833805 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:32.833904 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.834572 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.835545 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.835914 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.835981 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.835991 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.836121 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.836162 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:51:32.836217 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.833377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37517 (local address 127.0.98.2:41235)
0927 02:51:32.833430 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.833432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.833442 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:51:32.833472 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.833473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.833486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.833544 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:32.833715 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.834445 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.835463 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.835864 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.835913 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.835922 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.836115 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.836156 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:51:32.836210 20026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.833302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41235 (local address 127.0.98.3:37517)
0927 02:51:32.833353 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.833365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.833410 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.833554 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.833555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.833557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:32.833628 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.833631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.834512 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.834513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.835372 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.835376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.835938 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.835939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.836045 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.836048 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.836091 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.836132 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:51:32.836450 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.833309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46297 (local address 127.0.98.3:36849)
0927 02:51:32.833637 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.833646 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:32.833699 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:32.833805 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:32.833806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:32.833807 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:32.833856 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.833857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.834652 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.834653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.835453 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:32.835456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.835989 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.835990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:32.836054 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.836057 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:32.836106 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 02:51:32.836145 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":168,"thread_start_us":96,"threads_started":1}
I20260927 02:51:32.836612 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbbe0686d142a39d9f5570b4d5f7fe" candidate_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b8b850b81f434193cc0ce9234b1db9" is_pre_election: true
I20260927 02:51:32.836717 19710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbbe0686d142a39d9f5570b4d5f7fe" candidate_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" is_pre_election: true
I20260927 02:51:32.836786 19844 raft_consensus.cc:2454] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7b6e28f60145f5aea544ffb7c615ff in term 0.
I20260927 02:51:32.836841 19710 raft_consensus.cc:2454] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7b6e28f60145f5aea544ffb7c615ff in term 0.
I20260927 02:51:32.837034 19912 leader_election.cc:302] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7b6e28f60145f5aea544ffb7c615ff, 702a62fb1b904d5e8fc2d949a0b967a6; no voters:
I20260927 02:51:32.837204 20045 raft_consensus.cc:2781] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:51:32.837255 20045 raft_consensus.cc:482] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:51:32.837277 20045 raft_consensus.cc:3037] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:32.837970 20045 raft_consensus.cc:504] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.838094 20045 leader_election.cc:288] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [CANDIDATE]: Term 1 election: Requested vote from peers 20b8b850b81f434193cc0ce9234b1db9 (127.0.98.2:41235), 702a62fb1b904d5e8fc2d949a0b967a6 (127.0.98.1:46297)
I20260927 02:51:32.838263 19710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbbe0686d142a39d9f5570b4d5f7fe" candidate_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702a62fb1b904d5e8fc2d949a0b967a6"
I20260927 02:51:32.838263 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afbbe0686d142a39d9f5570b4d5f7fe" candidate_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b8b850b81f434193cc0ce9234b1db9"
I20260927 02:51:32.838340 19844 raft_consensus.cc:3037] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:32.838340 19710 raft_consensus.cc:3037] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:32.839197 19844 raft_consensus.cc:2454] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7b6e28f60145f5aea544ffb7c615ff in term 1.
I20260927 02:51:32.839197 19710 raft_consensus.cc:2454] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7b6e28f60145f5aea544ffb7c615ff in term 1.
I20260927 02:51:32.839361 19913 leader_election.cc:302] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b8b850b81f434193cc0ce9234b1db9, 4e7b6e28f60145f5aea544ffb7c615ff; no voters:
I20260927 02:51:32.839457 20045 raft_consensus.cc:2781] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:32.839588 20045 raft_consensus.cc:686] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 1 LEADER]: Becoming Leader. State: Replica: 4e7b6e28f60145f5aea544ffb7c615ff, State: Running, Role: LEADER
I20260927 02:51:32.839670 20045 consensus_queue.cc:237] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } }
I20260927 02:51:32.840462 19564 catalog_manager.cc:5495] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e7b6e28f60145f5aea544ffb7c615ff (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7b6e28f60145f5aea544ffb7c615ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 } health_report { overall_health: UNKNOWN } } }
I20260927 02:51:32.857049 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.854943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.0.98.3:36769)
0927 02:51:32.854998 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:32.855000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:32.855007 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:51:32.855072 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:32.855073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:32.855089 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:32.855142 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:32.855229 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.855682 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.856390 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:32.856563 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:32.856604 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:32.856716 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:32.856813 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:32.856867 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:32.856945 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:51:32.856960 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 02:51:32.858774 19844 raft_consensus.cc:1261] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 1 FOLLOWER]: Refusing update from remote peer 4e7b6e28f60145f5aea544ffb7c615ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:32.858839 19710 raft_consensus.cc:1261] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 1 FOLLOWER]: Refusing update from remote peer 4e7b6e28f60145f5aea544ffb7c615ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:32.859156 20045 consensus_queue.cc:1035] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "702a62fb1b904d5e8fc2d949a0b967a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:32.859294 20048 consensus_queue.cc:1035] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b8b850b81f434193cc0ce9234b1db9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:32.861375 20051 mvcc.cc:205] Tried to move back new op lower bound from 7333795810747047936 to 7333795810671943680. Current Snapshot: MvccSnapshot[applied={T|T < 7333795810747047936}]
I20260927 02:51:32.861821 20052 mvcc.cc:205] Tried to move back new op lower bound from 7333795810747047936 to 7333795810671943680. Current Snapshot: MvccSnapshot[applied={T|T < 7333795810747047936 or (T in {7333795810747047936})}]
I20260927 02:51:32.862013 20053 mvcc.cc:205] Tried to move back new op lower bound from 7333795810747047936 to 7333795810671943680. Current Snapshot: MvccSnapshot[applied={T|T < 7333795810747047936}]
I20260927 02:51:32.863878 19565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:32.863976 19565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:32.864684 19565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d558c213b3f6427cacc972888db3c300: Sending DeleteTablet for 3 replicas of tablet 7afbbe0686d142a39d9f5570b4d5f7fe
I20260927 02:51:32.864980 19824 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afbbe0686d142a39d9f5570b4d5f7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34078
I20260927 02:51:32.865018 19690 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afbbe0686d142a39d9f5570b4d5f7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54488
I20260927 02:51:32.865231 20061 tablet_replica.cc:330] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6: stopping tablet replica
I20260927 02:51:32.865324 20061 raft_consensus.cc:2229] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:32.865327 19958 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afbbe0686d142a39d9f5570b4d5f7fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51594
I20260927 02:51:32.865403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 19627
I20260927 02:51:32.865422 20061 raft_consensus.cc:2258] T 7afbbe0686d142a39d9f5570b4d5f7fe P 702a62fb1b904d5e8fc2d949a0b967a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:32.865504 20062 tablet_replica.cc:330] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: stopping tablet replica
I20260927 02:51:32.865579 20062 raft_consensus.cc:2229] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:51:32.865655 20062 raft_consensus.cc:2258] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:32.866110 20062 ts_tablet_manager.cc:1905] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:32.867741 20062 ts_tablet_manager.cc:1918] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:32.867811 20062 log.cc:1196] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/7afbbe0686d142a39d9f5570b4d5f7fe
I20260927 02:51:32.868074 20062 ts_tablet_manager.cc:1939] T 7afbbe0686d142a39d9f5570b4d5f7fe P 4e7b6e28f60145f5aea544ffb7c615ff: Deleting consensus metadata
I20260927 02:51:32.868443 20060 tablet_replica.cc:330] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: stopping tablet replica
I20260927 02:51:32.868484 19550 catalog_manager.cc:4841] TS 4e7b6e28f60145f5aea544ffb7c615ff (127.0.98.3:36769): tablet 7afbbe0686d142a39d9f5570b4d5f7fe (table test-table [id=8c7926e3dd8e44b0bc67ad3c8a91d3ac]) successfully deleted
I20260927 02:51:32.868549 20060 raft_consensus.cc:2229] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:32.868650 20060 raft_consensus.cc:2258] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:32.869110 20060 ts_tablet_manager.cc:1905] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:32.870580 20060 ts_tablet_manager.cc:1918] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:32.870649 20060 log.cc:1196] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/7afbbe0686d142a39d9f5570b4d5f7fe
W20260927 02:51:32.870899 19550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46297 (error 108)
I20260927 02:51:32.870926 20060 ts_tablet_manager.cc:1939] T 7afbbe0686d142a39d9f5570b4d5f7fe P 20b8b850b81f434193cc0ce9234b1db9: Deleting consensus metadata
I20260927 02:51:32.871301 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 19761
I20260927 02:51:32.871300 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:32.871178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46297 (local address 127.0.0.1:54504)
0927 02:51:32.871229 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:32.871266 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:51:32.871304 19551 catalog_manager.cc:4841] TS 20b8b850b81f434193cc0ce9234b1db9 (127.0.98.2:41235): tablet 7afbbe0686d142a39d9f5570b4d5f7fe (table test-table [id=8c7926e3dd8e44b0bc67ad3c8a91d3ac]) successfully deleted
W20260927 02:51:32.874954 19550 catalog_manager.cc:4584] TS 702a62fb1b904d5e8fc2d949a0b967a6 (127.0.98.1:46297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7afbbe0686d142a39d9f5570b4d5f7fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46297: connect: Connection refused (error 111)
I20260927 02:51:32.875473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 19895
I20260927 02:51:32.879352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 19533
2026-09-27T02:51:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1772 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T02:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:32Z Disabled control of system clock
I20260927 02:51:32.904104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:51:32.982347 20067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:32.982590 20067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:32.982611 20067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:32.983754 20067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:32.983793 20067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:32.983811 20067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:32.983822 20067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:32.983832 20067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:32.985145 20067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:32.985351 20067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:32.985600 20067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:32.990196 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:32.990206 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:32.990376 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:32.990414 20067 server_base.cc:1034] running on GCE node
I20260927 02:51:32.990649 20067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:32.990936 20067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:32.992107 20067 hybrid_clock.cc:648] HybridClock initialized: now 1790477492992093 us; error 30 us; skew 500 ppm
I20260927 02:51:32.993413 20067 webserver.cc:460] Webserver started at http://127.0.98.62:42355/ using document root <none> and password file <none>
I20260927 02:51:32.993659 20067 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:32.993712 20067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:32.993851 20067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:32.994942 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "ad00b5e851df45adb520700d96b4dd07"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "28cb730fe8b88122b9631354018ee72d"
server_key_iv: "fdd1fa2f96a49cc3754dbcebd74d45a4"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.995280 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad00b5e851df45adb520700d96b4dd07"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "28cb730fe8b88122b9631354018ee72d"
server_key_iv: "fdd1fa2f96a49cc3754dbcebd74d45a4"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.996511 20067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:32.997189 20081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:32.997335 20067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:32.997417 20067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "ad00b5e851df45adb520700d96b4dd07"
format_stamp: "Formatted at 2026-09-27 02:51:32 on dist-test-slave-f4rg"
server_key: "28cb730fe8b88122b9631354018ee72d"
server_key_iv: "fdd1fa2f96a49cc3754dbcebd74d45a4"
server_key_version: "encryptionkey@0"
I20260927 02:51:32.997488 20067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:33.018064 20067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:33.018399 20067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:33.018543 20067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:33.022464 20067 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42505
I20260927 02:51:33.022516 20133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42505 every 8 connection(s)
I20260927 02:51:33.022871 20067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:33.023784 20134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:33.026059 20134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07: Bootstrap starting.
I20260927 02:51:33.026624 20134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:33.026919 20134 log.cc:824] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:33.027616 20134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07: No bootstrap required, opened a new log
I20260927 02:51:33.028126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20067
I20260927 02:51:33.028240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:33.028437 392 external_mini_cluster.cc:1428] Setting key 02e15925c292ab089349397e2ba4cd07
I20260927 02:51:33.029616 20134 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad00b5e851df45adb520700d96b4dd07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260927 02:51:33.029754 20134 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:33.029783 20134 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad00b5e851df45adb520700d96b4dd07, State: Initialized, Role: FOLLOWER
I20260927 02:51:33.029910 20134 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad00b5e851df45adb520700d96b4dd07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260927 02:51:33.029991 20134 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:33.030027 20134 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:33.030074 20134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:33.030786 20134 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad00b5e851df45adb520700d96b4dd07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260927 02:51:33.030913 20134 leader_election.cc:302] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad00b5e851df45adb520700d96b4dd07; no voters:
I20260927 02:51:33.031068 20134 leader_election.cc:288] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:33.031109 20139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:33.031246 20139 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [term 1 LEADER]: Becoming Leader. State: Replica: ad00b5e851df45adb520700d96b4dd07, State: Running, Role: LEADER
I20260927 02:51:33.031319 20134 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:33.031333 20139 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad00b5e851df45adb520700d96b4dd07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } }
I20260927 02:51:33.031702 20140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad00b5e851df45adb520700d96b4dd07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad00b5e851df45adb520700d96b4dd07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260927 02:51:33.031778 20140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:33.031909 20141 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad00b5e851df45adb520700d96b4dd07. Latest consensus state: current_term: 1 leader_uuid: "ad00b5e851df45adb520700d96b4dd07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad00b5e851df45adb520700d96b4dd07" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42505 } } }
I20260927 02:51:33.031965 20141 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:33.032068 20146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:33.032572 20146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:33.034386 20146 catalog_manager.cc:1347] Generated new cluster ID: 1763b9174c02443abf79193aa158fb54
I20260927 02:51:33.034439 20146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:33.035226 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.028974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.0.98.62:42505)
0927 02:51:33.029173 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.029181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.029204 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:51:33.029598 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.029605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.029680 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.029812 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:33.030107 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.031894 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.033087 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.033669 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.033795 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.034505 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:33.034509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:33.034510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:33.034706 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:33.034746 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:33.034760 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:33.034847 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.034938 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.035013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260927 02:51:33.062700 20146 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:33.063298 20146 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:33.071574 20146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07: Generated new TSK 0
I20260927 02:51:33.071764 20146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:33.077566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--builtin_ntp_servers=127.0.98.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 02:51:33.159793 20158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:33.159999 20158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:33.160022 20158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:33.161167 20158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:33.161206 20158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:33.161232 20158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 02:51:33.162572 20158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:33.162791 20158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:33.163098 20158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:33.167733 20163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:33.167722 20164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:33.167913 20158 server_base.cc:1034] running on GCE node
W20260927 02:51:33.168090 20166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:33.168290 20158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:33.168561 20158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:33.169726 20158 hybrid_clock.cc:648] HybridClock initialized: now 1790477493169712 us; error 28 us; skew 500 ppm
I20260927 02:51:33.171036 20158 webserver.cc:460] Webserver started at http://127.0.98.1:33895/ using document root <none> and password file <none>
I20260927 02:51:33.171258 20158 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:33.171304 20158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:33.171424 20158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:33.172470 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data/instance:
uuid: "cdcd1e5ab868408c9a6c646fd42ddd52"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "019893ce1f087b3c27e17f305989faa8"
server_key_iv: "1fd09e53c8bef2fc96f8289a87bd65d1"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.172794 20158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cdcd1e5ab868408c9a6c646fd42ddd52"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "019893ce1f087b3c27e17f305989faa8"
server_key_iv: "1fd09e53c8bef2fc96f8289a87bd65d1"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.174014 20158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:33.174650 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.174847 20158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:33.174930 20158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/wal
uuid: "cdcd1e5ab868408c9a6c646fd42ddd52"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "019893ce1f087b3c27e17f305989faa8"
server_key_iv: "1fd09e53c8bef2fc96f8289a87bd65d1"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.174991 20158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:33.184916 20158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:33.185212 20158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:33.185344 20158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:33.185561 20158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:33.185881 20158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:33.185914 20158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.185942 20158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:33.185958 20158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.191453 20158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42101
I20260927 02:51:33.191510 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42101 every 8 connection(s)
I20260927 02:51:33.191816 20158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/data/info.pb
I20260927 02:51:33.192086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20158
I20260927 02:51:33.192202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-0/wal/instance
I20260927 02:51:33.192378 392 external_mini_cluster.cc:1428] Setting key 2bb2b9e4352251160dcb551a73a3d082
I20260927 02:51:33.193310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--builtin_ntp_servers=127.0.98.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:51:33.197641 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.192504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44071 (local address 127.0.98.62:42505)
0927 02:51:33.192549 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.192551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.192796 (+ 245us) server_negotiation.cc:407] Connection header received
0927 02:51:33.193468 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.193469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.193483 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.193556 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:33.194401 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.195155 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.196122 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.196405 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.196471 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.197375 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:33.197376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:33.197377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:33.197385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:33.197403 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:33.197408 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:33.197477 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.197524 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.197554 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:51:33.197844 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.192479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.1:44071)
0927 02:51:33.192671 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.192720 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.193338 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.193634 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.193639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.193650 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:33.194292 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.194299 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.195241 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.195242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.196023 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.196025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.196482 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.196482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.196619 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.197238 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:33.197238 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:33.197268 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:33.197291 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:33.197473 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:33.197475 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:33.197478 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.197522 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.197638 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260927 02:51:33.198263 20286 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260927 02:51:33.198345 20286 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:33.198509 20286 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260927 02:51:33.199007 20098 ts_manager.cc:194] Registered new tserver with Master: cdcd1e5ab868408c9a6c646fd42ddd52 (127.0.98.1:42101)
I20260927 02:51:33.199597 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44071
W20260927 02:51:33.273100 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:33.273326 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:33.273356 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:33.274530 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:33.274576 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:33.274617 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 02:51:33.275995 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:33.276240 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:33.276523 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:33.281327 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:33.281399 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:33.281543 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:33.281548 20289 server_base.cc:1034] running on GCE node
I20260927 02:51:33.281780 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:33.281999 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:33.283146 20289 hybrid_clock.cc:648] HybridClock initialized: now 1790477493283134 us; error 29 us; skew 500 ppm
I20260927 02:51:33.284373 20289 webserver.cc:460] Webserver started at http://127.0.98.2:34121/ using document root <none> and password file <none>
I20260927 02:51:33.284602 20289 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:33.284650 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:33.284775 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:33.285844 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data/instance:
uuid: "54ba4788e9ad499f858f2339761d64ff"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "5bd694e2a95d20056f4cf963ad44745c"
server_key_iv: "39bfd28306cff2359e8bebd9fbac9f5c"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.286176 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal/instance:
uuid: "54ba4788e9ad499f858f2339761d64ff"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "5bd694e2a95d20056f4cf963ad44745c"
server_key_iv: "39bfd28306cff2359e8bebd9fbac9f5c"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.287429 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:33.288096 20303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.288251 20289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:33.288331 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal
uuid: "54ba4788e9ad499f858f2339761d64ff"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "5bd694e2a95d20056f4cf963ad44745c"
server_key_iv: "39bfd28306cff2359e8bebd9fbac9f5c"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.288389 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:33.307103 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:33.307407 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:33.307545 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:33.307754 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:33.308071 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:33.308111 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.308140 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:33.308164 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.313409 20289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44055
I20260927 02:51:33.313473 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44055 every 8 connection(s)
I20260927 02:51:33.313783 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/data/info.pb
I20260927 02:51:33.318324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20289
I20260927 02:51:33.318424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal/instance
I20260927 02:51:33.318629 392 external_mini_cluster.cc:1428] Setting key 71fcbec883770a2f4566d349876e5e76
I20260927 02:51:33.319244 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.314436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59555 (local address 127.0.98.62:42505)
0927 02:51:33.314480 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.314482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.314706 (+ 224us) server_negotiation.cc:407] Connection header received
0927 02:51:33.315262 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.315263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.315274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.315326 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:33.315978 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.316484 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.317856 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.318050 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.318110 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.318945 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:33.318946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:33.318947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:33.318951 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:33.318970 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:33.318974 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:33.319025 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.319132 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.319160 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:33.319443 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.314430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.2:59555)
0927 02:51:33.314596 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.314630 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.315172 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.315382 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.315385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.315392 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:33.315874 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.315878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.316606 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.316607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.317761 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.317763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.318117 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.318118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.318227 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.318825 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:33.318825 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:33.318856 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:33.318879 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:33.319037 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:33.319044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:33.319048 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.319131 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.319241 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260927 02:51:33.319697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--builtin_ntp_servers=127.0.98.20:39055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 02:51:33.319929 20417 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260927 02:51:33.320016 20417 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:33.320185 20417 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260927 02:51:33.320505 20098 ts_manager.cc:194] Registered new tserver with Master: 54ba4788e9ad499f858f2339761d64ff (127.0.98.2:44055)
I20260927 02:51:33.320839 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:59555
W20260927 02:51:33.399124 20420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:33.399349 20420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:33.399371 20420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:33.400543 20420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:33.400606 20420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:33.400632 20420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 02:51:33.401988 20420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:33.402232 20420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:33.402519 20420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:33.407354 20425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:33.407367 20426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:33.407374 20428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:33.407423 20420 server_base.cc:1034] running on GCE node
I20260927 02:51:33.407811 20420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:33.408034 20420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:33.409168 20420 hybrid_clock.cc:648] HybridClock initialized: now 1790477493409156 us; error 38 us; skew 500 ppm
I20260927 02:51:33.410373 20420 webserver.cc:460] Webserver started at http://127.0.98.3:37713/ using document root <none> and password file <none>
I20260927 02:51:33.410591 20420 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:33.410638 20420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:33.410789 20420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:33.411855 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data/instance:
uuid: "47213fb4516a4ff5a76d3bda93cc6f80"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "f775db6c9d7655f46f37db8ab1553dc0"
server_key_iv: "6f110b6a6a637d0136106641c929c5d6"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.412184 20420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47213fb4516a4ff5a76d3bda93cc6f80"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "f775db6c9d7655f46f37db8ab1553dc0"
server_key_iv: "6f110b6a6a637d0136106641c929c5d6"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.413406 20420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:33.414130 20434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.414275 20420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:33.414355 20420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal
uuid: "47213fb4516a4ff5a76d3bda93cc6f80"
format_stamp: "Formatted at 2026-09-27 02:51:33 on dist-test-slave-f4rg"
server_key: "f775db6c9d7655f46f37db8ab1553dc0"
server_key_iv: "6f110b6a6a637d0136106641c929c5d6"
server_key_version: "encryptionkey@0"
I20260927 02:51:33.414422 20420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:33.427821 20420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:33.428117 20420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:33.428241 20420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:33.428447 20420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 02:51:33.428788 20420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 02:51:33.428822 20420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.428850 20420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 02:51:33.428875 20420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:33.434533 20420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40049
I20260927 02:51:33.434595 20547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40049 every 8 connection(s)
I20260927 02:51:33.434947 20420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/data/info.pb
I20260927 02:51:33.440063 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.435662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46221 (local address 127.0.98.62:42505)
0927 02:51:33.435722 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.435724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.435971 (+ 247us) server_negotiation.cc:407] Connection header received
0927 02:51:33.436481 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.436482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.436495 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.436547 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:33.437208 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.437706 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.438705 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.438956 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.439013 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.439799 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:33.439801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:33.439801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:33.439806 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:33.439821 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:33.439825 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:33.439871 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.439947 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.439981 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:51:33.440254 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.435652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42505 (local address 127.0.98.3:46221)
0927 02:51:33.435847 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.435889 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.436384 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.436629 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.436632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.436640 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:33.437122 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.437126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.437793 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.437794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.438620 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.438622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.439028 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.439029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.439151 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.439665 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 02:51:33.439666 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:33.439695 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 02:51:33.439718 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:33.439878 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:33.439883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:33.439886 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.439947 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.440052 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260927 02:51:33.440711 20548 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42505
I20260927 02:51:33.440799 20548 heartbeater.cc:464] Registering TS with master...
I20260927 02:51:33.440975 20548 heartbeater.cc:511] Master 127.0.98.62:42505 requested a full tablet report, sending...
I20260927 02:51:33.441314 20098 ts_manager.cc:194] Registered new tserver with Master: 47213fb4516a4ff5a76d3bda93cc6f80 (127.0.98.3:40049)
I20260927 02:51:33.441702 20098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46221
I20260927 02:51:33.444442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20420
I20260927 02:51:33.444540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal/instance
I20260927 02:51:33.444746 392 external_mini_cluster.cc:1428] Setting key dd5ff146b75c7fde451df1a09b7f17ea
I20260927 02:51:33.445565 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 02:51:33.449467 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.446847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.98.62:42505)
0927 02:51:33.446903 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.446905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.447025 (+ 120us) server_negotiation.cc:407] Connection header received
0927 02:51:33.447062 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.447062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.447073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.447117 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:33.447249 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.447726 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.448488 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.448661 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.448704 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.449295 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:33.449296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:33.449296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:33.449299 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:33.449309 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:33.449312 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:33.449343 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.449376 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.449396 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 02:51:33.451025 20098 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 02:51:33.452464 20098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 02:51:33.458622 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.454589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34186 (local address 127.0.98.3:40049)
0927 02:51:33.454896 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.454900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.454993 (+ 93us) server_negotiation.cc:407] Connection header received
0927 02:51:33.455076 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.455078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.455101 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.455172 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:33.455398 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.456528 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.457716 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.458215 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.458277 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.458379 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.458498 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.458536 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260927 02:51:33.458611 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.454494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40049 (local address 127.0.0.1:34186)
0927 02:51:33.454891 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.454933 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.455001 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.455251 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.455254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.455257 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:33.455325 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.455327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.456621 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.456622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.457645 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.457647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.458292 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.458293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.458383 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.458399 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.458465 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.458528 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":80,"threads_started":1}
I20260927 02:51:33.458742 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.454668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35686 (local address 127.0.98.1:42101)
0927 02:51:33.454866 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.454871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.455084 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:51:33.455155 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.455157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.455172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.455237 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:33.455433 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.456150 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.457491 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.458250 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.458317 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.458404 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.458620 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.458659 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260927 02:51:33.458788 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.454848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.0.98.2:44055)
0927 02:51:33.455005 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.455009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.455222 (+ 213us) server_negotiation.cc:407] Connection header received
0927 02:51:33.455262 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.455265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.455279 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.455343 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:33.455508 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.456576 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.457541 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.458250 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.458317 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.458410 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.458668 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.458715 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260927 02:51:33.458925 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.454714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44055 (local address 127.0.0.1:60738)
0927 02:51:33.455104 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.455112 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.455154 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.455408 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.455408 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.455410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:33.455444 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.455445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.456662 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.456663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.457460 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.457462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.458443 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.458444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.458526 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.458532 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.458576 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.458626 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":3,"thread_start_us":64,"threads_started":1}
I20260927 02:51:33.458990 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.454494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42101 (local address 127.0.0.1:35686)
0927 02:51:33.455021 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.455031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.455090 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.455308 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.455309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.455311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:33.455358 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.455359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.456224 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.456224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.457368 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.457371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.458441 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.458442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.458526 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.458530 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.458848 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.458890 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":24,"thread_start_us":81,"threads_started":1}
I20260927 02:51:33.459622 20351 tablet_service.cc:1467] Processing CreateTablet for tablet 873e3a4d74084d45a794fb49a505df2c (DEFAULT_TABLE table=test-table [id=9548a1de18c546f0a96b9fd386767149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:33.459854 20220 tablet_service.cc:1467] Processing CreateTablet for tablet 873e3a4d74084d45a794fb49a505df2c (DEFAULT_TABLE table=test-table [id=9548a1de18c546f0a96b9fd386767149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:33.459995 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873e3a4d74084d45a794fb49a505df2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:33.459929 20482 tablet_service.cc:1467] Processing CreateTablet for tablet 873e3a4d74084d45a794fb49a505df2c (DEFAULT_TABLE table=test-table [id=9548a1de18c546f0a96b9fd386767149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 02:51:33.460208 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873e3a4d74084d45a794fb49a505df2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:33.460198 20220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873e3a4d74084d45a794fb49a505df2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:33.462993 20562 tablet_bootstrap.cc:492] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Bootstrap starting.
I20260927 02:51:33.463395 20563 tablet_bootstrap.cc:492] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Bootstrap starting.
I20260927 02:51:33.463644 20562 tablet_bootstrap.cc:654] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:33.463625 20564 tablet_bootstrap.cc:492] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52: Bootstrap starting.
I20260927 02:51:33.463968 20562 log.cc:824] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:33.464248 20564 tablet_bootstrap.cc:654] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:33.464505 20564 log.cc:824] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:33.464668 20562 tablet_bootstrap.cc:492] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: No bootstrap required, opened a new log
I20260927 02:51:33.464744 20562 ts_tablet_manager.cc:1397] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:33.465234 20564 tablet_bootstrap.cc:492] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52: No bootstrap required, opened a new log
I20260927 02:51:33.465247 20563 tablet_bootstrap.cc:654] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:33.465306 20564 ts_tablet_manager.cc:1397] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 02:51:33.465538 20563 log.cc:824] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:33.466859 20562 raft_consensus.cc:348] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.467038 20562 raft_consensus.cc:374] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:33.467085 20562 raft_consensus.cc:729] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54ba4788e9ad499f858f2339761d64ff, State: Initialized, Role: FOLLOWER
I20260927 02:51:33.467223 20562 consensus_queue.cc:260] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.467346 20563 tablet_bootstrap.cc:492] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: No bootstrap required, opened a new log
I20260927 02:51:33.467407 20563 ts_tablet_manager.cc:1397] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 02:51:33.467319 20564 raft_consensus.cc:348] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.467489 20564 raft_consensus.cc:374] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:33.467504 20562 ts_tablet_manager.cc:1428] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:33.467530 20564 raft_consensus.cc:729] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdcd1e5ab868408c9a6c646fd42ddd52, State: Initialized, Role: FOLLOWER
I20260927 02:51:33.467621 20417 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260927 02:51:33.467641 20564 consensus_queue.cc:260] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.467878 20564 ts_tablet_manager.cc:1428] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:33.467976 20286 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260927 02:51:33.469476 20563 raft_consensus.cc:348] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.469601 20563 raft_consensus.cc:374] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:33.469622 20563 raft_consensus.cc:729] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47213fb4516a4ff5a76d3bda93cc6f80, State: Initialized, Role: FOLLOWER
I20260927 02:51:33.469709 20563 consensus_queue.cc:260] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.469950 20563 ts_tablet_manager.cc:1428] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 02:51:33.470001 20548 heartbeater.cc:503] Master 127.0.98.62:42505 was elected leader, sending a full tablet report...
I20260927 02:51:33.542722 20568 raft_consensus.cc:482] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 02:51:33.542889 20568 raft_consensus.cc:504] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.543242 20568 leader_election.cc:288] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47213fb4516a4ff5a76d3bda93cc6f80 (127.0.98.3:40049), cdcd1e5ab868408c9a6c646fd42ddd52 (127.0.98.1:42101)
I20260927 02:51:33.545729 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.543408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42101 (local address 127.0.98.2:37205)
0927 02:51:33.543448 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.543460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.543508 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.543659 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.543659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.543661 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:33.543725 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.543727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.544305 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.544306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.545017 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.545018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.545508 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.545509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.545570 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.545573 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.545617 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.545657 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 02:51:33.545758 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.543496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37205 (local address 127.0.98.1:42101)
0927 02:51:33.543542 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.543544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.543553 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:33.543581 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.543582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.543603 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.543644 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:33.543785 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.544239 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.545095 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.545445 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.545504 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.545513 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.545663 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.545701 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 02:51:33.545969 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.543530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34313 (local address 127.0.98.3:40049)
0927 02:51:33.543594 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.543596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.543699 (+ 103us) server_negotiation.cc:407] Connection header received
0927 02:51:33.543753 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.543754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.543762 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.543805 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 02:51:33.543933 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.544388 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.545255 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.545607 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.545664 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.545673 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.545834 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.545877 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 02:51:33.546059 20240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873e3a4d74084d45a794fb49a505df2c" candidate_uuid: "54ba4788e9ad499f858f2339761d64ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" is_pre_election: true
I20260927 02:51:33.546175 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.543423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40049 (local address 127.0.98.2:34313)
0927 02:51:33.543632 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.543641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:33.543697 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:33.543814 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:33.543814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:33.543815 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 02:51:33.543880 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.543881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.544454 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.544455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.545178 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:33.545180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.545673 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.545674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:33.545756 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.545759 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 02:51:33.546044 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 02:51:33.546092 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
I20260927 02:51:33.546213 20240 raft_consensus.cc:2454] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54ba4788e9ad499f858f2339761d64ff in term 0.
I20260927 02:51:33.546422 20306 leader_election.cc:302] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54ba4788e9ad499f858f2339761d64ff, cdcd1e5ab868408c9a6c646fd42ddd52; no voters:
I20260927 02:51:33.546507 20502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873e3a4d74084d45a794fb49a505df2c" candidate_uuid: "54ba4788e9ad499f858f2339761d64ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" is_pre_election: true
I20260927 02:51:33.546566 20568 raft_consensus.cc:2781] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 02:51:33.546608 20568 raft_consensus.cc:482] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 02:51:33.546636 20568 raft_consensus.cc:3037] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:33.546669 20502 raft_consensus.cc:2454] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54ba4788e9ad499f858f2339761d64ff in term 0.
I20260927 02:51:33.547400 20568 raft_consensus.cc:504] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.547534 20568 leader_election.cc:288] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [CANDIDATE]: Term 1 election: Requested vote from peers 47213fb4516a4ff5a76d3bda93cc6f80 (127.0.98.3:40049), cdcd1e5ab868408c9a6c646fd42ddd52 (127.0.98.1:42101)
I20260927 02:51:33.547664 20502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873e3a4d74084d45a794fb49a505df2c" candidate_uuid: "54ba4788e9ad499f858f2339761d64ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47213fb4516a4ff5a76d3bda93cc6f80"
I20260927 02:51:33.547681 20240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873e3a4d74084d45a794fb49a505df2c" candidate_uuid: "54ba4788e9ad499f858f2339761d64ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52"
I20260927 02:51:33.547737 20502 raft_consensus.cc:3037] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:33.547744 20240 raft_consensus.cc:3037] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:33.548390 20502 raft_consensus.cc:2454] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54ba4788e9ad499f858f2339761d64ff in term 1.
I20260927 02:51:33.548444 20240 raft_consensus.cc:2454] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54ba4788e9ad499f858f2339761d64ff in term 1.
I20260927 02:51:33.548575 20306 leader_election.cc:302] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54ba4788e9ad499f858f2339761d64ff, cdcd1e5ab868408c9a6c646fd42ddd52; no voters:
I20260927 02:51:33.548673 20568 raft_consensus.cc:2781] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:33.548802 20568 raft_consensus.cc:686] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 1 LEADER]: Becoming Leader. State: Replica: 54ba4788e9ad499f858f2339761d64ff, State: Running, Role: LEADER
I20260927 02:51:33.548883 20568 consensus_queue.cc:237] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } }
I20260927 02:51:33.549630 20098 catalog_manager.cc:5495] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 54ba4788e9ad499f858f2339761d64ff (127.0.98.2). New cstate: current_term: 1 leader_uuid: "54ba4788e9ad499f858f2339761d64ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54ba4788e9ad499f858f2339761d64ff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44055 } health_report { overall_health: HEALTHY } } }
W20260927 02:51:33.564296 20418 tablet.cc:2369] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 02:51:33.573585 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.571338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.0.98.2:44055)
0927 02:51:33.571397 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:33.571399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:33.571408 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:33.571453 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:33.571454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:33.571472 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:33.571554 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:33.571705 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.572168 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.572882 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:33.573074 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:33.573128 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:33.573269 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:33.573371 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:33.573444 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:33.573468 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 02:51:33.573486 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:51:33.575379 20240 raft_consensus.cc:1261] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 1 FOLLOWER]: Refusing update from remote peer 54ba4788e9ad499f858f2339761d64ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:33.575380 20502 raft_consensus.cc:1261] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 1 FOLLOWER]: Refusing update from remote peer 54ba4788e9ad499f858f2339761d64ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:33.575717 20572 consensus_queue.cc:1035] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdcd1e5ab868408c9a6c646fd42ddd52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:33.575846 20568 consensus_queue.cc:1035] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "47213fb4516a4ff5a76d3bda93cc6f80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:33.578691 20576 mvcc.cc:205] Tried to move back new op lower bound from 7333795813682085888 to 7333795813576851456. Current Snapshot: MvccSnapshot[applied={T|T < 7333795813682085888}]
I20260927 02:51:33.579897 20578 mvcc.cc:205] Tried to move back new op lower bound from 7333795813682085888 to 7333795813576851456. Current Snapshot: MvccSnapshot[applied={T|T < 7333795813682085888 or (T in {7333795813682085888})}]
I20260927 02:51:33.580010 20098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:33.580103 20098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 02:51:33.580838 20098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad00b5e851df45adb520700d96b4dd07: Sending DeleteTablet for 3 replicas of tablet 873e3a4d74084d45a794fb49a505df2c
I20260927 02:51:33.581087 20351 tablet_service.cc:1514] Processing DeleteTablet for tablet 873e3a4d74084d45a794fb49a505df2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:33 UTC) from {username='slave'} at 127.0.0.1:60738
I20260927 02:51:33.581261 20482 tablet_service.cc:1514] Processing DeleteTablet for tablet 873e3a4d74084d45a794fb49a505df2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:33 UTC) from {username='slave'} at 127.0.0.1:34186
I20260927 02:51:33.581305 20585 tablet_replica.cc:330] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: stopping tablet replica
I20260927 02:51:33.581425 20481 tablet_service.cc:1514] Processing DeleteTablet for tablet 873e3a4d74084d45a794fb49a505df2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:33 UTC) from {username='slave'} at 127.0.0.1:34186
I20260927 02:51:33.581471 20586 tablet_replica.cc:330] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: stopping tablet replica
I20260927 02:51:33.581449 20585 raft_consensus.cc:2229] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 1 LEADER]: Raft consensus shutting down.
I20260927 02:51:33.581542 20220 tablet_service.cc:1514] Processing DeleteTablet for tablet 873e3a4d74084d45a794fb49a505df2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 02:51:33 UTC) from {username='slave'} at 127.0.0.1:35686
I20260927 02:51:33.581596 20586 raft_consensus.cc:2229] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:33.581697 20586 raft_consensus.cc:2258] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:33.581718 20585 raft_consensus.cc:2258] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 02:51:33.581997 20587 tablet_replica.cc:330] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52: stopping tablet replica
I20260927 02:51:33.582077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20158
I20260927 02:51:33.582091 20587 raft_consensus.cc:2229] T 873e3a4d74084d45a794fb49a505df2c P cdcd1e5ab868408c9a6c646fd42ddd52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 02:51:33.582405 20585 ts_tablet_manager.cc:1905] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:33.582588 20586 ts_tablet_manager.cc:1905] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 02:51:33.584311 20585 ts_tablet_manager.cc:1918] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:33.584383 20585 log.cc:1196] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-1/wal/wals/873e3a4d74084d45a794fb49a505df2c
I20260927 02:51:33.584620 20585 ts_tablet_manager.cc:1939] T 873e3a4d74084d45a794fb49a505df2c P 54ba4788e9ad499f858f2339761d64ff: Deleting consensus metadata
I20260927 02:51:33.585778 20586 ts_tablet_manager.cc:1918] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 02:51:33.585858 20586 log.cc:1196] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Deleting WAL directory at /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790477232139228-392-0/minicluster-data/ts-2/wal/wals/873e3a4d74084d45a794fb49a505df2c
I20260927 02:51:33.586099 20586 ts_tablet_manager.cc:1939] T 873e3a4d74084d45a794fb49a505df2c P 47213fb4516a4ff5a76d3bda93cc6f80: Deleting consensus metadata
I20260927 02:51:33.586648 20083 catalog_manager.cc:4841] TS 47213fb4516a4ff5a76d3bda93cc6f80 (127.0.98.3:40049): tablet 873e3a4d74084d45a794fb49a505df2c (table test-table [id=9548a1de18c546f0a96b9fd386767149]) successfully deleted
W20260927 02:51:33.586794 20083 catalog_manager.cc:4808] TS 47213fb4516a4ff5a76d3bda93cc6f80 (127.0.98.3:40049): delete failed for tablet 873e3a4d74084d45a794fb49a505df2c because the tablet was not found. No further retry: Not found: Tablet not found: 873e3a4d74084d45a794fb49a505df2c
I20260927 02:51:33.586887 20083 catalog_manager.cc:4841] TS 54ba4788e9ad499f858f2339761d64ff (127.0.98.2:44055): tablet 873e3a4d74084d45a794fb49a505df2c (table test-table [id=9548a1de18c546f0a96b9fd386767149]) successfully deleted
W20260927 02:51:33.587553 20084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42101 (error 108)
I20260927 02:51:33.587985 20556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:33.587865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42101 (local address 127.0.0.1:35696)
0927 02:51:33.587916 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:33.587953 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:51:33.587998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20289
W20260927 02:51:33.590235 20084 catalog_manager.cc:4584] TS cdcd1e5ab868408c9a6c646fd42ddd52 (127.0.98.1:42101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 873e3a4d74084d45a794fb49a505df2c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42101: connect: Connection refused (error 111)
I20260927 02:51:33.593060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20420
I20260927 02:51:33.596678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20067
2026-09-27T02:51:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (717 ms)
[----------] 6 tests from EncryptionPolicyTest (8223 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:33 dist-test-slave-f4rg krb5kdc[20591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:33 dist-test-slave-f4rg krb5kdc[20591](info): set up 2 sockets
Sep 27 02:51:33 dist-test-slave-f4rg krb5kdc[20591](info): commencing operation
krb5kdc: starting...
W20260927 02:51:34.542435 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:34 dist-test-slave-f4rg krb5kdc[20591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:34.651935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:34.728746 20604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:34.728935 20604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:34.728956 20604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:34.730070 20604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:34.730110 20604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:34.730123 20604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:34.730134 20604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:34.730146 20604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:34.731587 20604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:34.731798 20604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:34.732007 20604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:34.736459 20609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:34.736483 20610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:34.736482 20612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:34.736593 20604 server_base.cc:1034] running on GCE node
I20260927 02:51:34.736909 20604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:34.737142 20604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:34.738294 20604 hybrid_clock.cc:648] HybridClock initialized: now 1790477494738280 us; error 28 us; skew 500 ppm
Sep 27 02:51:34 dist-test-slave-f4rg krb5kdc[20591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:34.740574 20604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:34.740983 20604 webserver.cc:460] Webserver started at http://127.0.0.1:41245/ using document root <none> and password file <none>
I20260927 02:51:34.741185 20604 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:34.741231 20604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:34.741351 20604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:34.742345 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "9882e18038904d1eb57223f6d9cb4077"
format_stamp: "Formatted at 2026-09-27 02:51:34 on dist-test-slave-f4rg"
server_key: "c5ccebbf0ad98f1636e9eb638b769f7c"
server_key_iv: "8f0a8bec22e4a88d70437d6249e80290"
server_key_version: "encryptionkey@0"
I20260927 02:51:34.742653 20604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "9882e18038904d1eb57223f6d9cb4077"
format_stamp: "Formatted at 2026-09-27 02:51:34 on dist-test-slave-f4rg"
server_key: "c5ccebbf0ad98f1636e9eb638b769f7c"
server_key_iv: "8f0a8bec22e4a88d70437d6249e80290"
server_key_version: "encryptionkey@0"
I20260927 02:51:34.743876 20604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:34.744607 20619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:34.744786 20604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:34.744855 20604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "9882e18038904d1eb57223f6d9cb4077"
format_stamp: "Formatted at 2026-09-27 02:51:34 on dist-test-slave-f4rg"
server_key: "c5ccebbf0ad98f1636e9eb638b769f7c"
server_key_iv: "8f0a8bec22e4a88d70437d6249e80290"
server_key_version: "encryptionkey@0"
I20260927 02:51:34.744921 20604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:34.757898 20604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:34.759750 20604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:34.759881 20604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:34.763521 20604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44715
I20260927 02:51:34.763578 20671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44715 every 8 connection(s)
I20260927 02:51:34.763871 20604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:34.764642 20672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:34.766119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20604
I20260927 02:51:34.766239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:34.766422 392 external_mini_cluster.cc:1428] Setting key efe6c19520f3a53c1cc3c149a15cb556
I20260927 02:51:34.767138 20672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077: Bootstrap starting.
I20260927 02:51:34.767678 20672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:34.767966 20672 log.cc:824] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:34.768627 20672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077: No bootstrap required, opened a new log
I20260927 02:51:34.770612 20672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9882e18038904d1eb57223f6d9cb4077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260927 02:51:34.770740 20672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:34.770759 20672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9882e18038904d1eb57223f6d9cb4077, State: Initialized, Role: FOLLOWER
I20260927 02:51:34.770888 20672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9882e18038904d1eb57223f6d9cb4077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260927 02:51:34.770962 20672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:34.770996 20672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:34.771044 20672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:34.771638 20672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9882e18038904d1eb57223f6d9cb4077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260927 02:51:34.771739 20672 leader_election.cc:302] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9882e18038904d1eb57223f6d9cb4077; no voters:
I20260927 02:51:34.771878 20672 leader_election.cc:288] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:34.771924 20677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:34.772078 20677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [term 1 LEADER]: Becoming Leader. State: Replica: 9882e18038904d1eb57223f6d9cb4077, State: Running, Role: LEADER
I20260927 02:51:34.772105 20672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:34.772161 20677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9882e18038904d1eb57223f6d9cb4077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } }
I20260927 02:51:34.772480 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9882e18038904d1eb57223f6d9cb4077. Latest consensus state: current_term: 1 leader_uuid: "9882e18038904d1eb57223f6d9cb4077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9882e18038904d1eb57223f6d9cb4077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } } }
I20260927 02:51:34.772547 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [sys.catalog]: This master's current role is: LEADER
Sep 27 02:51:34 dist-test-slave-f4rg krb5kdc[20591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:51:34.773061 20684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:34.773424 20678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9882e18038904d1eb57223f6d9cb4077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9882e18038904d1eb57223f6d9cb4077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44715 } } }
I20260927 02:51:34.773501 20684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:34.773571 20678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:34.774976 20684 catalog_manager.cc:1347] Generated new cluster ID: 05768a6cafb84cc9925a516b3236ba6e
I20260927 02:51:34.775030 20684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:34.776932 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:34.766932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50319 (local address 127.0.0.1:44715)
0927 02:51:34.767081 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:34.767089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:34.767113 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:51:34.767566 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:34.767573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:34.767622 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:34.767750 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:34.768514 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:34.769409 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:34.770432 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:34.770860 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:34.770957 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:34.773625 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:34.773629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:34.773630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:34.773641 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:34.775540 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:34.775880 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:34.775884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:34.775884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:34.775966 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:34.776156 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:34.776157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:34.776157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:34.776375 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:34.776519 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:34.776642 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 02:51:34.776746 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260927 02:51:34.783233 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:34.792412 20675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:34.784463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47247 (local address 127.0.0.1:44715)
0927 02:51:34.784503 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:34.784505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:34.786967 (+ 2462us) server_negotiation.cc:407] Connection header received
0927 02:51:34.787022 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:34.787023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:34.787032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:34.787083 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:34.787577 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:34.788064 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:34.788723 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:34.788938 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:34.788990 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:34.790076 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:34.790077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:34.790077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:34.790084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:34.791311 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:34.791622 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:34.791623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:34.791623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:34.791681 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:34.791891 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:34.791891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:34.791892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:34.792014 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:34.792074 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:34.792262 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 02:51:34.792320 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:51:34.798790 20684 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:34.799497 20684 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:34.810468 20684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9882e18038904d1eb57223f6d9cb4077: Generated new TSK 0
I20260927 02:51:34.810674 20684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:34.818569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20604
W20260927 02:51:34.822376 20589 connection.cc:535] client connection to 127.0.0.1:44715 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44715): Connection reset by peer (error 104)
2026-09-27T02:51:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:34 dist-test-slave-f4rg krb5kdc[20703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:34 dist-test-slave-f4rg krb5kdc[20703](info): set up 2 sockets
Sep 27 02:51:34 dist-test-slave-f4rg krb5kdc[20703](info): commencing operation
krb5kdc: starting...
W20260927 02:51:35.758534 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:35 dist-test-slave-f4rg krb5kdc[20703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:35.867686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35545
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:35.947063 20716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:35.947263 20716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:35.947285 20716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:35.948398 20716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:35.948431 20716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:35.948443 20716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:35.948454 20716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:35.948468 20716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:35.949834 20716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35545
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:35.950012 20716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:35.950178 20716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:35.954447 20721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:35.954602 20724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:35.954682 20716 server_base.cc:1034] running on GCE node
W20260927 02:51:35.954448 20722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:35.954963 20716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:35.955233 20716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:35.956365 20716 hybrid_clock.cc:648] HybridClock initialized: now 1790477495956358 us; error 30 us; skew 500 ppm
Sep 27 02:51:35 dist-test-slave-f4rg krb5kdc[20703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:35.961666 20716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:35.962059 20716 webserver.cc:460] Webserver started at http://127.0.0.1:38055/ using document root <none> and password file <none>
I20260927 02:51:35.962244 20716 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:35.962288 20716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:35.962407 20716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:35.963426 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "1def1e3fd6ad4a548f36b3bfcf66564a"
format_stamp: "Formatted at 2026-09-27 02:51:35 on dist-test-slave-f4rg"
server_key: "606020f34e097bc5db84f013c0d2c75f"
server_key_iv: "7d186d2670c83198a9f2648cef21b9be"
server_key_version: "encryptionkey@0"
I20260927 02:51:35.963740 20716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "1def1e3fd6ad4a548f36b3bfcf66564a"
format_stamp: "Formatted at 2026-09-27 02:51:35 on dist-test-slave-f4rg"
server_key: "606020f34e097bc5db84f013c0d2c75f"
server_key_iv: "7d186d2670c83198a9f2648cef21b9be"
server_key_version: "encryptionkey@0"
I20260927 02:51:35.964980 20716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:35.965747 20731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:35.965909 20716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:35.965993 20716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "1def1e3fd6ad4a548f36b3bfcf66564a"
format_stamp: "Formatted at 2026-09-27 02:51:35 on dist-test-slave-f4rg"
server_key: "606020f34e097bc5db84f013c0d2c75f"
server_key_iv: "7d186d2670c83198a9f2648cef21b9be"
server_key_version: "encryptionkey@0"
I20260927 02:51:35.966054 20716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:35.977104 20716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:35.978915 20716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:35.979050 20716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:35.982648 20716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35545
I20260927 02:51:35.982683 20783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35545 every 8 connection(s)
I20260927 02:51:35.983011 20716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:35.983829 20784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:35.986060 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a: Bootstrap starting.
I20260927 02:51:35.986618 20784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:35.986904 20784 log.cc:824] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:35.987543 20784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a: No bootstrap required, opened a new log
I20260927 02:51:35.989113 20784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35545 } }
I20260927 02:51:35.989267 20784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:35.989295 20784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1def1e3fd6ad4a548f36b3bfcf66564a, State: Initialized, Role: FOLLOWER
I20260927 02:51:35.989430 20784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35545 } }
I20260927 02:51:35.989528 20784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:35.989554 20784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:35.989584 20784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:35.990124 20784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35545 } }
I20260927 02:51:35.990221 20784 leader_election.cc:302] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1def1e3fd6ad4a548f36b3bfcf66564a; no voters:
I20260927 02:51:35.990389 20784 leader_election.cc:288] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:35.990430 20787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:35.990592 20787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [term 1 LEADER]: Becoming Leader. State: Replica: 1def1e3fd6ad4a548f36b3bfcf66564a, State: Running, Role: LEADER
I20260927 02:51:35.990621 20784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:35.990692 20787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35545 } }
I20260927 02:51:35.991097 20789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1def1e3fd6ad4a548f36b3bfcf66564a. Latest consensus state: current_term: 1 leader_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35545 } } }
I20260927 02:51:35.991191 20789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:35.991277 20788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1def1e3fd6ad4a548f36b3bfcf66564a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35545 } } }
I20260927 02:51:35.991370 20788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:35.991678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20716
I20260927 02:51:35.991797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:35.991988 392 external_mini_cluster.cc:1428] Setting key 4a4a0ad9642351eff1aeda39eaf8ed75
I20260927 02:51:35.991968 20796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:35.992486 20796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:35.993919 20796 catalog_manager.cc:1347] Generated new cluster ID: d65e535893d348c9b27fb505d95ed922
I20260927 02:51:35.993968 20796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:35.998395 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:35.992439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53687 (local address 127.0.0.1:35545)
0927 02:51:35.993106 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:35.993116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:35.993150 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:51:35.993261 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:35.993268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:35.993330 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:35.993474 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:35.993814 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:35.995200 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:35.996284 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:35.996919 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:35.997036 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:35.997734 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:35.997739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:35.997739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:35.997944 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:35.997983 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:35.997998 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:35.998086 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:35.998153 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 02:51:35.998181 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":69,"threads_started":1}
I20260927 02:51:35.999307 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:36.004190 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:36.000781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47913 (local address 127.0.0.1:35545)
0927 02:51:36.000813 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:36.000814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:36.000931 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:51:36.000993 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:36.000994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:36.001013 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:36.001065 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:36.001179 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:36.001892 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:36.002942 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:36.003235 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:36.003288 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:36.003950 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:36.003951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:36.003952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:36.003956 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:36.003971 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:36.003975 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:36.004024 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:36.004100 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 02:51:36.004113 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:51:36.007511 20796 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:36.008245 20796 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:36.022078 20796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1def1e3fd6ad4a548f36b3bfcf66564a: Generated new TSK 0
I20260927 02:51:36.022323 20796 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:36.024138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20716
W20260927 02:51:36.027756 20701 connection.cc:535] client connection to 127.0.0.1:35545 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35545): Connection reset by peer (error 104)
2026-09-27T02:51:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:36 dist-test-slave-f4rg krb5kdc[20815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:36 dist-test-slave-f4rg krb5kdc[20815](info): set up 2 sockets
Sep 27 02:51:36 dist-test-slave-f4rg krb5kdc[20815](info): commencing operation
krb5kdc: starting...
W20260927 02:51:36.974547 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:37 dist-test-slave-f4rg krb5kdc[20815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:37.087204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:37.164855 20828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:37.165079 20828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:37.165113 20828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:37.166594 20828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:37.166644 20828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:37.166662 20828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:37.166678 20828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:37.166702 20828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:37.168713 20828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:37.168969 20828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:37.169176 20828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:37.173801 20833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:37.173825 20834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:37.173801 20836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:37.173969 20828 server_base.cc:1034] running on GCE node
I20260927 02:51:37.174221 20828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:37.174463 20828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:37.175609 20828 hybrid_clock.cc:648] HybridClock initialized: now 1790477497175597 us; error 31 us; skew 500 ppm
Sep 27 02:51:37 dist-test-slave-f4rg krb5kdc[20815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:37.177959 20828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:37.178378 20828 webserver.cc:460] Webserver started at http://127.0.0.1:38101/ using document root <none> and password file <none>
I20260927 02:51:37.178566 20828 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:37.178612 20828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:37.178699 20828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:37.179776 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "c5915181a1c04ba898dbf759d5b4c1e6"
format_stamp: "Formatted at 2026-09-27 02:51:37 on dist-test-slave-f4rg"
server_key: "1b7f8cf0d03b05cc6519d6dba037c1f4"
server_key_iv: "89a36e597b5c330e897e045de8a559ef"
server_key_version: "encryptionkey@0"
I20260927 02:51:37.180065 20828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "c5915181a1c04ba898dbf759d5b4c1e6"
format_stamp: "Formatted at 2026-09-27 02:51:37 on dist-test-slave-f4rg"
server_key: "1b7f8cf0d03b05cc6519d6dba037c1f4"
server_key_iv: "89a36e597b5c330e897e045de8a559ef"
server_key_version: "encryptionkey@0"
I20260927 02:51:37.181242 20828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:37.182003 20843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:37.182202 20828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:37.182291 20828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "c5915181a1c04ba898dbf759d5b4c1e6"
format_stamp: "Formatted at 2026-09-27 02:51:37 on dist-test-slave-f4rg"
server_key: "1b7f8cf0d03b05cc6519d6dba037c1f4"
server_key_iv: "89a36e597b5c330e897e045de8a559ef"
server_key_version: "encryptionkey@0"
I20260927 02:51:37.182348 20828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:37.185551 20828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:37.187777 20828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:37.187888 20828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:37.191581 20828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42791
I20260927 02:51:37.191617 20895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42791 every 8 connection(s)
I20260927 02:51:37.191924 20828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:37.192873 20896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:37.195197 20896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6: Bootstrap starting.
I20260927 02:51:37.195732 20896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:37.195993 20896 log.cc:824] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:37.196647 20896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6: No bootstrap required, opened a new log
I20260927 02:51:37.198264 20896 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260927 02:51:37.198400 20896 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:37.198431 20896 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5915181a1c04ba898dbf759d5b4c1e6, State: Initialized, Role: FOLLOWER
I20260927 02:51:37.198570 20896 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260927 02:51:37.198644 20896 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:37.198684 20896 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:37.198730 20896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:37.199270 20896 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260927 02:51:37.199364 20896 leader_election.cc:302] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5915181a1c04ba898dbf759d5b4c1e6; no voters:
I20260927 02:51:37.199517 20896 leader_election.cc:288] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:37.199575 20899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:37.199752 20896 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:37.199898 20899 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [term 1 LEADER]: Becoming Leader. State: Replica: c5915181a1c04ba898dbf759d5b4c1e6, State: Running, Role: LEADER
I20260927 02:51:37.200027 20899 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } }
I20260927 02:51:37.200364 20900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } } }
I20260927 02:51:37.200438 20900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:37.200590 20901 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5915181a1c04ba898dbf759d5b4c1e6. Latest consensus state: current_term: 1 leader_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5915181a1c04ba898dbf759d5b4c1e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42791 } } }
I20260927 02:51:37.200644 20901 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:37.200945 20908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:37.201267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20828
I20260927 02:51:37.201375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:37.201468 20908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:37.201581 392 external_mini_cluster.cc:1428] Setting key 3155a6dafa112fe64f33fcf18a1debde
I20260927 02:51:37.203389 20908 catalog_manager.cc:1347] Generated new cluster ID: 66113860b8df4daa9a14baa70b1abb47
I20260927 02:51:37.203445 20908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:37.204069 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:37.201984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50643 (local address 127.0.0.1:42791)
0927 02:51:37.202133 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:37.202142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:37.202338 (+ 196us) server_negotiation.cc:407] Connection header received
0927 02:51:37.202480 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:37.202488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:37.202544 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:37.202688 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:37.203558 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:37.203562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:37.203562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:37.203567 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:37.203608 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:37.203622 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:37.203671 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:37.203866 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 02:51:37.203904 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260927 02:51:37.204762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:37.207320 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:37.206176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58347 (local address 127.0.0.1:42791)
0927 02:51:37.206258 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:37.206260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:37.206372 (+ 112us) server_negotiation.cc:407] Connection header received
0927 02:51:37.206433 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:37.206434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:37.206441 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:37.206497 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:37.207122 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:37.207123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:37.207123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:37.207126 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:37.207141 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:37.207145 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:37.207188 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:37.207252 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 02:51:37.207264 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 02:51:37.213624 20908 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:37.213807 20908 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:37.220234 20908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5915181a1c04ba898dbf759d5b4c1e6: Generated new TSK 0
I20260927 02:51:37.220415 20908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:37.224593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20828
2026-09-27T02:51:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:37 dist-test-slave-f4rg krb5kdc[20927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:37 dist-test-slave-f4rg krb5kdc[20927](info): set up 2 sockets
Sep 27 02:51:37 dist-test-slave-f4rg krb5kdc[20927](info): commencing operation
krb5kdc: starting...
W20260927 02:51:38.171069 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:38 dist-test-slave-f4rg krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:38.281633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:38.360940 20940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:38.361150 20940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:38.361181 20940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:38.362329 20940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:38.362370 20940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:38.362389 20940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:38.362404 20940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:38.362419 20940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:38.363924 20940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:38.364171 20940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:38.364375 20940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:38.369077 20946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:38.369087 20948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:38.369104 20940 server_base.cc:1034] running on GCE node
W20260927 02:51:38.369174 20945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:38.369472 20940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:38.369750 20940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:38.370864 20940 hybrid_clock.cc:648] HybridClock initialized: now 1790477498370842 us; error 31 us; skew 500 ppm
Sep 27 02:51:38 dist-test-slave-f4rg krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:38.373122 20940 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:38.373539 20940 webserver.cc:460] Webserver started at http://127.0.0.1:33717/ using document root <none> and password file <none>
I20260927 02:51:38.373729 20940 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:38.373770 20940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:38.373883 20940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:38.374930 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "c9e9805a25bf4e1db0999342ec6ddd8f"
format_stamp: "Formatted at 2026-09-27 02:51:38 on dist-test-slave-f4rg"
server_key: "e4771d5951a97be2eb0892496ab89c4d"
server_key_iv: "73e9967ec3cb0f005213c2ed7099cc47"
server_key_version: "encryptionkey@0"
I20260927 02:51:38.375248 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9e9805a25bf4e1db0999342ec6ddd8f"
format_stamp: "Formatted at 2026-09-27 02:51:38 on dist-test-slave-f4rg"
server_key: "e4771d5951a97be2eb0892496ab89c4d"
server_key_iv: "73e9967ec3cb0f005213c2ed7099cc47"
server_key_version: "encryptionkey@0"
I20260927 02:51:38.376421 20940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:38.377024 20955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:38.377167 20940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:38.377250 20940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "c9e9805a25bf4e1db0999342ec6ddd8f"
format_stamp: "Formatted at 2026-09-27 02:51:38 on dist-test-slave-f4rg"
server_key: "e4771d5951a97be2eb0892496ab89c4d"
server_key_iv: "73e9967ec3cb0f005213c2ed7099cc47"
server_key_version: "encryptionkey@0"
I20260927 02:51:38.377303 20940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:38.389454 20940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:38.391534 20940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:38.391649 20940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:38.395228 20940 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42269
I20260927 02:51:38.395262 21007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42269 every 8 connection(s)
I20260927 02:51:38.395573 20940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:38.395913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 20940
I20260927 02:51:38.396023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:38.396215 392 external_mini_cluster.cc:1428] Setting key ce5d37737b8351c8c122b8634092b667
I20260927 02:51:38.396493 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:38.399763 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f: Bootstrap starting.
I20260927 02:51:38.400327 21008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:38.400627 21008 log.cc:824] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:38.401305 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f: No bootstrap required, opened a new log
Sep 27 02:51:38 dist-test-slave-f4rg krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:51:38.403496 21008 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260927 02:51:38.403663 21008 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:38.403721 21008 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9805a25bf4e1db0999342ec6ddd8f, State: Initialized, Role: FOLLOWER
I20260927 02:51:38.403848 21008 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260927 02:51:38.403923 21008 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:38.403952 21008 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:38.403997 21008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:38.404629 21008 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260927 02:51:38.404742 21008 leader_election.cc:302] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9e9805a25bf4e1db0999342ec6ddd8f; no voters:
I20260927 02:51:38.404898 21008 leader_election.cc:288] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:38.404947 21013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:38.405110 21013 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [term 1 LEADER]: Becoming Leader. State: Replica: c9e9805a25bf4e1db0999342ec6ddd8f, State: Running, Role: LEADER
I20260927 02:51:38.405201 21008 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:38.405215 21013 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } }
I20260927 02:51:38.405525 21013 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e9805a25bf4e1db0999342ec6ddd8f. Latest consensus state: current_term: 1 leader_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } } }
I20260927 02:51:38.405715 21013 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:38.405658 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e9805a25bf4e1db0999342ec6ddd8f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42269 } } }
I20260927 02:51:38.406066 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:38.406257 21022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:38.406765 21022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:38.407037 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:38.396661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58375 (local address 127.0.0.1:42269)
0927 02:51:38.396842 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:38.396850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:38.396881 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:51:38.397373 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:38.397383 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:38.397448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:38.397597 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:38.398125 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:38.399483 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:38.400390 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:38.400940 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:38.401022 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:38.403419 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:38.403423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:38.403424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:38.403436 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:38.405232 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:38.405625 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:38.405628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:38.405628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:38.405733 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:38.405951 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:38.405952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:38.405952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:38.406261 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:38.406440 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:38.406695 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 02:51:38.406851 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260927 02:51:38.407914 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:38.409032 21022 catalog_manager.cc:1347] Generated new cluster ID: a9bdf69e5a64437490c2047a2c46cea3
I20260927 02:51:38.409073 21022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:38.414585 21022 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:38.415307 21022 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:38.417871 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:38.409161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40461 (local address 127.0.0.1:42269)
0927 02:51:38.409202 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:38.409203 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:38.409430 (+ 227us) server_negotiation.cc:407] Connection header received
0927 02:51:38.409460 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:38.409461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:38.409471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:38.409535 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:38.410083 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:38.410826 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:38.411754 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:38.412064 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:38.412127 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:38.415312 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:38.415314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:38.415314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:38.415321 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:38.416806 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:38.417160 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:38.417161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:38.417162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:38.417218 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:38.417376 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:38.417377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:38.417377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:38.417504 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:38.417591 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:38.417694 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 02:51:38.417779 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:51:38.419963 21022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9e9805a25bf4e1db0999342ec6ddd8f: Generated new TSK 0
I20260927 02:51:38.420138 21022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:38.428011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 20940
2026-09-27T02:51:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:38 dist-test-slave-f4rg krb5kdc[21039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:38 dist-test-slave-f4rg krb5kdc[21039](info): set up 2 sockets
Sep 27 02:51:38 dist-test-slave-f4rg krb5kdc[21039](info): commencing operation
krb5kdc: starting...
W20260927 02:51:39.374511 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:39 dist-test-slave-f4rg krb5kdc[21039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:39.482888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:39.560570 21052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:39.560786 21052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:39.560817 21052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:39.561939 21052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:39.561980 21052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:39.561997 21052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:39.562014 21052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:39.562029 21052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:39.563468 21052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:39.563706 21052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:39.563899 21052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:39.568446 21058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:39.568559 21052 server_base.cc:1034] running on GCE node
W20260927 02:51:39.568460 21060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:39.568526 21057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:39.568907 21052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:39.569149 21052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:39.570282 21052 hybrid_clock.cc:648] HybridClock initialized: now 1790477499570262 us; error 33 us; skew 500 ppm
Sep 27 02:51:39 dist-test-slave-f4rg krb5kdc[21039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:39.572587 21052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:39.572985 21052 webserver.cc:460] Webserver started at http://127.0.0.1:39305/ using document root <none> and password file <none>
I20260927 02:51:39.573190 21052 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:39.573236 21052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:39.573357 21052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:39.574422 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "bef4309d4747426387e8338ee1870ae0"
format_stamp: "Formatted at 2026-09-27 02:51:39 on dist-test-slave-f4rg"
server_key: "4131f4245a9398ae0058d090cd156b95"
server_key_iv: "04399206eb212e5567f1b30517317049"
server_key_version: "encryptionkey@0"
I20260927 02:51:39.574740 21052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "bef4309d4747426387e8338ee1870ae0"
format_stamp: "Formatted at 2026-09-27 02:51:39 on dist-test-slave-f4rg"
server_key: "4131f4245a9398ae0058d090cd156b95"
server_key_iv: "04399206eb212e5567f1b30517317049"
server_key_version: "encryptionkey@0"
I20260927 02:51:39.575923 21052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 02:51:39.576545 21067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:39.576689 21052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:39.576759 21052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "bef4309d4747426387e8338ee1870ae0"
format_stamp: "Formatted at 2026-09-27 02:51:39 on dist-test-slave-f4rg"
server_key: "4131f4245a9398ae0058d090cd156b95"
server_key_iv: "04399206eb212e5567f1b30517317049"
server_key_version: "encryptionkey@0"
I20260927 02:51:39.576820 21052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:39.602037 21052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:39.604218 21052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:39.604351 21052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:39.607959 21052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40071
I20260927 02:51:39.607992 21119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40071 every 8 connection(s)
I20260927 02:51:39.608305 21052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:39.609083 21120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:39.611241 21120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0: Bootstrap starting.
I20260927 02:51:39.611732 21120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:39.611943 21120 log.cc:824] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:39.612526 21120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0: No bootstrap required, opened a new log
I20260927 02:51:39.613977 21120 raft_consensus.cc:348] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4309d4747426387e8338ee1870ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40071 } }
I20260927 02:51:39.614105 21120 raft_consensus.cc:374] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:39.614125 21120 raft_consensus.cc:729] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef4309d4747426387e8338ee1870ae0, State: Initialized, Role: FOLLOWER
I20260927 02:51:39.614249 21120 consensus_queue.cc:260] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4309d4747426387e8338ee1870ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40071 } }
I20260927 02:51:39.614320 21120 raft_consensus.cc:388] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:39.614347 21120 raft_consensus.cc:482] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:39.614379 21120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:39.614835 21120 raft_consensus.cc:504] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4309d4747426387e8338ee1870ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40071 } }
I20260927 02:51:39.614912 21120 leader_election.cc:302] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bef4309d4747426387e8338ee1870ae0; no voters:
I20260927 02:51:39.615023 21120 leader_election.cc:288] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:39.615068 21123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:39.615223 21120 sys_catalog.cc:564] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:39.615231 21123 raft_consensus.cc:686] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [term 1 LEADER]: Becoming Leader. State: Replica: bef4309d4747426387e8338ee1870ae0, State: Running, Role: LEADER
I20260927 02:51:39.615331 21123 consensus_queue.cc:237] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4309d4747426387e8338ee1870ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40071 } }
I20260927 02:51:39.615700 21125 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bef4309d4747426387e8338ee1870ae0. Latest consensus state: current_term: 1 leader_uuid: "bef4309d4747426387e8338ee1870ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4309d4747426387e8338ee1870ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40071 } } }
I20260927 02:51:39.615715 21124 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bef4309d4747426387e8338ee1870ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef4309d4747426387e8338ee1870ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40071 } } }
I20260927 02:51:39.615851 21125 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:39.615868 21124 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:39.616236 21133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:39.616750 21133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:39.616846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21052
I20260927 02:51:39.616978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:39.617178 392 external_mini_cluster.cc:1428] Setting key 6b1bde0e70b9b2842a72fabae73f41bf
I20260927 02:51:39.618260 21133 catalog_manager.cc:1347] Generated new cluster ID: 833b8e3c9a46449387e06f0251f7286e
I20260927 02:51:39.618309 21133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:39.623168 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:39.617593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45801 (local address 127.0.0.1:40071)
0927 02:51:39.617750 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:39.617759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:39.617981 (+ 222us) server_negotiation.cc:407] Connection header received
0927 02:51:39.618102 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:39.618110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:39.618165 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:39.618317 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:39.618646 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:39.620024 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:39.621113 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:39.621684 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:39.621771 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:39.622426 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:39.622430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:39.622430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:39.622615 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:39.622652 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:39.622665 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:39.622718 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:39.622788 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 02:51:39.622968 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260927 02:51:39.623999 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:39.628753 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:39.625369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55297 (local address 127.0.0.1:40071)
0927 02:51:39.625406 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:39.625408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:39.625643 (+ 235us) server_negotiation.cc:407] Connection header received
0927 02:51:39.625714 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:39.625715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:39.625724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:39.625748 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:39.625894 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:39.626582 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:39.627558 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:39.627850 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:39.627883 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:39.628538 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:39.628539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:39.628539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:39.628543 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:39.628555 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:39.628559 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:39.628580 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:39.628661 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:51:39.628683 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 02:51:39.639842 21133 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:39.640331 21133 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:39.648394 21133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bef4309d4747426387e8338ee1870ae0: Generated new TSK 0
I20260927 02:51:39.648553 21133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:39.650719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21052
2026-09-27T02:51:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:39 dist-test-slave-f4rg krb5kdc[21151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:39 dist-test-slave-f4rg krb5kdc[21151](info): set up 2 sockets
Sep 27 02:51:39 dist-test-slave-f4rg krb5kdc[21151](info): commencing operation
krb5kdc: starting...
W20260927 02:51:40.586434 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:40 dist-test-slave-f4rg krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:40.693295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:40.771231 21164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:40.771426 21164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:40.771447 21164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:40.772550 21164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:40.772590 21164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:40.772611 21164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:40.772624 21164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:40.772648 21164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:40.774070 21164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:40.774283 21164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:40.774482 21164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:40.778838 21169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:40.778956 21164 server_base.cc:1034] running on GCE node
W20260927 02:51:40.778838 21172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:40.778877 21170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:40.779255 21164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:40.779505 21164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:40.780654 21164 hybrid_clock.cc:648] HybridClock initialized: now 1790477500780636 us; error 31 us; skew 500 ppm
Sep 27 02:51:40 dist-test-slave-f4rg krb5kdc[21151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:40.783042 21164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:40.783465 21164 webserver.cc:460] Webserver started at http://127.0.0.1:38415/ using document root <none> and password file <none>
I20260927 02:51:40.783664 21164 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:40.783710 21164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:40.783833 21164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:40.784862 21164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "fdf0d749f9a44061928160270e97037c"
format_stamp: "Formatted at 2026-09-27 02:51:40 on dist-test-slave-f4rg"
server_key: "76fec529e1bae41079c13de79a2a447b"
server_key_iv: "028e92c4d1dce8b83fa8de2a5126d96a"
server_key_version: "encryptionkey@0"
I20260927 02:51:40.785181 21164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdf0d749f9a44061928160270e97037c"
format_stamp: "Formatted at 2026-09-27 02:51:40 on dist-test-slave-f4rg"
server_key: "76fec529e1bae41079c13de79a2a447b"
server_key_iv: "028e92c4d1dce8b83fa8de2a5126d96a"
server_key_version: "encryptionkey@0"
I20260927 02:51:40.786330 21164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:40.787015 21179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:40.787165 21164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:40.787250 21164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "fdf0d749f9a44061928160270e97037c"
format_stamp: "Formatted at 2026-09-27 02:51:40 on dist-test-slave-f4rg"
server_key: "76fec529e1bae41079c13de79a2a447b"
server_key_iv: "028e92c4d1dce8b83fa8de2a5126d96a"
server_key_version: "encryptionkey@0"
I20260927 02:51:40.787314 21164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:40.790416 21164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:40.792467 21164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:40.792596 21164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:40.796278 21164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44807
I20260927 02:51:40.796308 21231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44807 every 8 connection(s)
I20260927 02:51:40.796593 21164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:40.797359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21164
I20260927 02:51:40.797474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:40.797492 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:40.797657 392 external_mini_cluster.cc:1428] Setting key 5cd4ef03cb90ce3a53eb17cdb0006e51
I20260927 02:51:40.799954 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:40.798136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43211 (local address 127.0.0.1:44807)
0927 02:51:40.798291 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:40.798300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:40.798331 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:51:40.798767 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:40.798776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:40.798829 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:40.798973 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:40.799519 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:40.799523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:40.799523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:40.799528 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:40.799568 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:40.799582 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:40.799654 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:40.799733 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:51:40.799776 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260927 02:51:40.800848 21232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c: Bootstrap starting.
I20260927 02:51:40.800864 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:40.801380 21232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:40.801628 21232 log.cc:824] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:40.802325 21232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c: No bootstrap required, opened a new log
I20260927 02:51:40.803763 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:40.802925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38795 (local address 127.0.0.1:44807)
0927 02:51:40.802975 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:40.802977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:40.802986 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:40.803004 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:40.803005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:40.803008 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:40.803066 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:40.803571 (+ 505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:40.803572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:40.803572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:40.803576 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:40.803590 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:40.803594 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:40.803654 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:40.803687 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 02:51:40.803700 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 02:51:40.804352 21232 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0d749f9a44061928160270e97037c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260927 02:51:40.804486 21232 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:40.804507 21232 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf0d749f9a44061928160270e97037c, State: Initialized, Role: FOLLOWER
I20260927 02:51:40.804618 21232 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0d749f9a44061928160270e97037c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260927 02:51:40.804690 21232 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:40.804725 21232 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:40.804771 21232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:40.805261 21232 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0d749f9a44061928160270e97037c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260927 02:51:40.805348 21232 leader_election.cc:302] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf0d749f9a44061928160270e97037c; no voters:
I20260927 02:51:40.805506 21232 leader_election.cc:288] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:40.805545 21242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:40.805699 21242 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [term 1 LEADER]: Becoming Leader. State: Replica: fdf0d749f9a44061928160270e97037c, State: Running, Role: LEADER
I20260927 02:51:40.805735 21232 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:40.805780 21242 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0d749f9a44061928160270e97037c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260927 02:51:40.806092 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf0d749f9a44061928160270e97037c. Latest consensus state: current_term: 1 leader_uuid: "fdf0d749f9a44061928160270e97037c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0d749f9a44061928160270e97037c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260927 02:51:40.806123 21244 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf0d749f9a44061928160270e97037c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf0d749f9a44061928160270e97037c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260927 02:51:40.806169 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:40.806178 21244 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:40.806358 21248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:40.806878 21248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:40.808305 21248 catalog_manager.cc:1347] Generated new cluster ID: a2a590f2ad324bf0a3c930142d349e9c
I20260927 02:51:40.808344 21248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:40.831360 21248 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:40.831528 21248 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:40.833799 21248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf0d749f9a44061928160270e97037c: Generated new TSK 0
I20260927 02:51:40.833954 21248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:40.839278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21164
2026-09-27T02:51:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1188 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:40 dist-test-slave-f4rg krb5kdc[21263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:40 dist-test-slave-f4rg krb5kdc[21263](info): set up 2 sockets
Sep 27 02:51:40 dist-test-slave-f4rg krb5kdc[21263](info): commencing operation
krb5kdc: starting...
W20260927 02:51:41.770514 392 mini_kdc.cc:121] Time spent starting KDC: real 0.914s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:41 dist-test-slave-f4rg krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:41.877843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:41.956139 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:41.956334 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:41.956364 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:41.957448 21276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:41.957487 21276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:41.957506 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:41.957521 21276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:41.957538 21276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:41.959007 21276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:41.959244 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:41.959436 21276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:41.963830 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:41.963817 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:41.964025 21276 server_base.cc:1034] running on GCE node
W20260927 02:51:41.963828 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:41.964291 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:41.964550 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:41.965675 21276 hybrid_clock.cc:648] HybridClock initialized: now 1790477501965661 us; error 28 us; skew 500 ppm
Sep 27 02:51:41 dist-test-slave-f4rg krb5kdc[21263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:41.970846 21276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:41.971259 21276 webserver.cc:460] Webserver started at http://127.0.0.1:44291/ using document root <none> and password file <none>
I20260927 02:51:41.971431 21276 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:41.971469 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:41.971581 21276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:41.972599 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "610db00a7dba429983c4516c6c6dcc70"
format_stamp: "Formatted at 2026-09-27 02:51:41 on dist-test-slave-f4rg"
server_key: "03f8e09567541646dc90dfaa9812dd22"
server_key_iv: "d72fbadcd79701f3b2babfe90d922fe0"
server_key_version: "encryptionkey@0"
I20260927 02:51:41.972908 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "610db00a7dba429983c4516c6c6dcc70"
format_stamp: "Formatted at 2026-09-27 02:51:41 on dist-test-slave-f4rg"
server_key: "03f8e09567541646dc90dfaa9812dd22"
server_key_iv: "d72fbadcd79701f3b2babfe90d922fe0"
server_key_version: "encryptionkey@0"
I20260927 02:51:41.974082 21276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:41.974728 21291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:41.974895 21276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:41.974973 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "610db00a7dba429983c4516c6c6dcc70"
format_stamp: "Formatted at 2026-09-27 02:51:41 on dist-test-slave-f4rg"
server_key: "03f8e09567541646dc90dfaa9812dd22"
server_key_iv: "d72fbadcd79701f3b2babfe90d922fe0"
server_key_version: "encryptionkey@0"
I20260927 02:51:41.975025 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:41.994498 21276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:41.996268 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:41.996377 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:41.999981 21276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43735
I20260927 02:51:42.000058 21343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43735 every 8 connection(s)
I20260927 02:51:42.000349 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:42.001170 21344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:42.001951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21276
I20260927 02:51:42.002053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:42.002225 392 external_mini_cluster.cc:1428] Setting key 29d2cabf4d7e3c6cf6baf580b238f708
I20260927 02:51:42.003978 21344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70: Bootstrap starting.
I20260927 02:51:42.004541 21344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:42.004792 21344 log.cc:824] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:42.005378 21344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70: No bootstrap required, opened a new log
I20260927 02:51:42.007311 21344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610db00a7dba429983c4516c6c6dcc70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260927 02:51:42.007443 21344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:42.007488 21344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610db00a7dba429983c4516c6c6dcc70, State: Initialized, Role: FOLLOWER
I20260927 02:51:42.007617 21344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610db00a7dba429983c4516c6c6dcc70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260927 02:51:42.007694 21344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:42.007721 21344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:42.007766 21344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 02:51:42 dist-test-slave-f4rg krb5kdc[21263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:51:42.008417 21344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610db00a7dba429983c4516c6c6dcc70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260927 02:51:42.008512 21344 leader_election.cc:302] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 610db00a7dba429983c4516c6c6dcc70; no voters:
I20260927 02:51:42.008680 21344 leader_election.cc:288] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:42.008718 21349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:42.008870 21349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [term 1 LEADER]: Becoming Leader. State: Replica: 610db00a7dba429983c4516c6c6dcc70, State: Running, Role: LEADER
I20260927 02:51:42.008932 21344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:42.008961 21349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610db00a7dba429983c4516c6c6dcc70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } }
I20260927 02:51:42.009294 21349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 610db00a7dba429983c4516c6c6dcc70. Latest consensus state: current_term: 1 leader_uuid: "610db00a7dba429983c4516c6c6dcc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610db00a7dba429983c4516c6c6dcc70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } } }
I20260927 02:51:42.009415 21349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:42.009466 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "610db00a7dba429983c4516c6c6dcc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "610db00a7dba429983c4516c6c6dcc70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43735 } } }
I20260927 02:51:42.009601 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:42.009689 21357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:42.010187 21357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:42.012344 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:42.002719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:44625 (local address 127.0.0.1:43735)
0927 02:51:42.002899 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:42.002907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:42.002933 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:51:42.003336 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:42.003342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:42.003391 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:42.003513 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:42.004124 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:42.005073 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:42.005825 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:42.006235 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:42.006331 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:42.008847 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:42.008852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:42.008853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:42.008865 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:42.010922 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:42.011319 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:42.011322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:42.011323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:42.011405 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:42.011598 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:42.011599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:42.011599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:42.011827 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:42.011954 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:42.012087 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 02:51:42.012183 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260927 02:51:42.013237 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:42.016031 21357 catalog_manager.cc:1347] Generated new cluster ID: e5a86f203608466eb0805036d8ea8fb5
I20260927 02:51:42.016080 21357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:42.021008 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:42.014403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:46281 (local address 127.0.0.1:43735)
0927 02:51:42.014435 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:42.014437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:42.014567 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:51:42.014621 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:42.014622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:42.014631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:42.014674 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:42.015180 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:42.015749 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:42.016740 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:42.017036 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:42.017093 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:42.018430 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:42.018431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:42.018432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:42.018438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:42.019915 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:42.020301 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:42.020302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:42.020302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:42.020349 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:42.020545 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:42.020545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:42.020546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:42.020652 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:42.020734 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:42.020868 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 02:51:42.020922 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:51:42.038376 21357 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:42.039256 21357 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:42.048583 21357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 610db00a7dba429983c4516c6c6dcc70: Generated new TSK 0
I20260927 02:51:42.048825 21357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:42.060101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21276
W20260927 02:51:42.063998 21261 connection.cc:535] client connection to 127.0.0.1:43735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43735): Connection reset by peer (error 104)
2026-09-27T02:51:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:42 dist-test-slave-f4rg krb5kdc[21375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:42 dist-test-slave-f4rg krb5kdc[21375](info): set up 2 sockets
Sep 27 02:51:42 dist-test-slave-f4rg krb5kdc[21375](info): commencing operation
krb5kdc: starting...
W20260927 02:51:43.002650 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:43 dist-test-slave-f4rg krb5kdc[21375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:43.110405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:43.188448 21388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:43.188679 21388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:43.188710 21388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:43.190080 21388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:43.190133 21388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:43.190155 21388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:43.190169 21388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:43.190186 21388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:43.191742 21388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:43.191980 21388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:43.192180 21388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:43.196688 21393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:43.196703 21396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:43.196729 21394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:43.196913 21388 server_base.cc:1034] running on GCE node
I20260927 02:51:43.197146 21388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:43.197386 21388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:43.198550 21388 hybrid_clock.cc:648] HybridClock initialized: now 1790477503198525 us; error 39 us; skew 500 ppm
Sep 27 02:51:43 dist-test-slave-f4rg krb5kdc[21375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:43.201009 21388 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:43.201442 21388 webserver.cc:460] Webserver started at http://127.0.0.1:37821/ using document root <none> and password file <none>
I20260927 02:51:43.201633 21388 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:43.201670 21388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:43.201781 21388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:43.202761 21388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "01086c5ebc1a48c69ed50567278f1f06"
format_stamp: "Formatted at 2026-09-27 02:51:43 on dist-test-slave-f4rg"
server_key: "16273aca9db5b731d74b7749321a77c3"
server_key_iv: "65071e2e3bd9ebc5803137eb6bf706c5"
server_key_version: "encryptionkey@0"
I20260927 02:51:43.203096 21388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "01086c5ebc1a48c69ed50567278f1f06"
format_stamp: "Formatted at 2026-09-27 02:51:43 on dist-test-slave-f4rg"
server_key: "16273aca9db5b731d74b7749321a77c3"
server_key_iv: "65071e2e3bd9ebc5803137eb6bf706c5"
server_key_version: "encryptionkey@0"
I20260927 02:51:43.204299 21388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:43.205000 21403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:43.205147 21388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:43.205221 21388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "01086c5ebc1a48c69ed50567278f1f06"
format_stamp: "Formatted at 2026-09-27 02:51:43 on dist-test-slave-f4rg"
server_key: "16273aca9db5b731d74b7749321a77c3"
server_key_iv: "65071e2e3bd9ebc5803137eb6bf706c5"
server_key_version: "encryptionkey@0"
I20260927 02:51:43.205293 21388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:43.231204 21388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:43.233001 21388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:43.233139 21388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:43.236747 21388 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45351
I20260927 02:51:43.236794 21455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45351 every 8 connection(s)
I20260927 02:51:43.237102 21388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:43.237872 21456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:43.240130 21456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06: Bootstrap starting.
I20260927 02:51:43.240674 21456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:43.240921 21456 log.cc:824] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:43.241554 21456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06: No bootstrap required, opened a new log
I20260927 02:51:43.243079 21456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01086c5ebc1a48c69ed50567278f1f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260927 02:51:43.243220 21456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:43.243242 21456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01086c5ebc1a48c69ed50567278f1f06, State: Initialized, Role: FOLLOWER
I20260927 02:51:43.243377 21456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01086c5ebc1a48c69ed50567278f1f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260927 02:51:43.243449 21456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:43.243482 21456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:43.243530 21456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:43.243992 21456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01086c5ebc1a48c69ed50567278f1f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260927 02:51:43.244079 21456 leader_election.cc:302] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01086c5ebc1a48c69ed50567278f1f06; no voters:
I20260927 02:51:43.244230 21456 leader_election.cc:288] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:43.244267 21459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:43.244403 21459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [term 1 LEADER]: Becoming Leader. State: Replica: 01086c5ebc1a48c69ed50567278f1f06, State: Running, Role: LEADER
I20260927 02:51:43.244470 21456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:43.244489 21459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01086c5ebc1a48c69ed50567278f1f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260927 02:51:43.244779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21388
I20260927 02:51:43.244899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:43.245121 392 external_mini_cluster.cc:1428] Setting key 3c0d10e0b79f9d1bfd615d6318305de9
I20260927 02:51:43.245572 21460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01086c5ebc1a48c69ed50567278f1f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01086c5ebc1a48c69ed50567278f1f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260927 02:51:43.245673 21460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:43.245958 21461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01086c5ebc1a48c69ed50567278f1f06. Latest consensus state: current_term: 1 leader_uuid: "01086c5ebc1a48c69ed50567278f1f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01086c5ebc1a48c69ed50567278f1f06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260927 02:51:43.246027 21461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06 [sys.catalog]: This master's current role is: LEADER
W20260927 02:51:43.246197 21475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:51:43.246238 21475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:43.246336 21476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:43.246526 21476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:43.247957 21476 catalog_manager.cc:1347] Generated new cluster ID: a6968ffbf34742f78e013708f34d7d11
I20260927 02:51:43.248008 21476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:43.250515 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:43.245765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:52633 (local address 127.0.0.1:45351)
0927 02:51:43.246427 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:43.246437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:43.246468 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:51:43.246571 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:43.246579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:43.246639 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:43.246768 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:43.246993 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:43.247959 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:43.248774 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:43.249209 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:43.249302 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:43.249937 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:43.249941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:43.249941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:43.250102 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:43.250131 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:43.250142 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:43.250207 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:43.250301 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 02:51:43.250328 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":59,"threads_started":1}
I20260927 02:51:43.251346 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:43.253896 21476 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:43.254575 21476 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:43.261541 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:43.253737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:50187 (local address 127.0.0.1:45351)
0927 02:51:43.253770 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:43.253772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:43.258947 (+ 5175us) server_negotiation.cc:407] Connection header received
0927 02:51:43.259007 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:43.259008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:43.259019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:43.259067 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:43.259198 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:43.259736 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:43.260429 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:43.260636 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:43.260696 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:43.261316 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:43.261318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:43.261318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:43.261322 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:43.261338 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:43.261343 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:43.261384 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:43.261449 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:51:43.261461 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:51:43.263695 21476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01086c5ebc1a48c69ed50567278f1f06: Generated new TSK 0
I20260927 02:51:43.263851 21476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:43.267767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21388
W20260927 02:51:43.271276 21373 connection.cc:535] client connection to 127.0.0.1:45351 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45351): Connection reset by peer (error 104)
2026-09-27T02:51:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:43 dist-test-slave-f4rg krb5kdc[21487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:43 dist-test-slave-f4rg krb5kdc[21487](info): set up 2 sockets
Sep 27 02:51:43 dist-test-slave-f4rg krb5kdc[21487](info): commencing operation
krb5kdc: starting...
W20260927 02:51:44.234549 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:44 dist-test-slave-f4rg krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:44.343827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:44.421200 21500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:44.421403 21500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:44.421425 21500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:44.422552 21500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:44.422595 21500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:44.422608 21500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:44.422619 21500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:44.422629 21500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:44.424093 21500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:44.424315 21500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:44.424518 21500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:44.429092 21505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:44.429186 21500 server_base.cc:1034] running on GCE node
W20260927 02:51:44.429291 21506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:44.429370 21508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:44.429580 21500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:44.429883 21500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:44.431015 21500 hybrid_clock.cc:648] HybridClock initialized: now 1790477504430997 us; error 34 us; skew 500 ppm
Sep 27 02:51:44 dist-test-slave-f4rg krb5kdc[21487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:44.433274 21500 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:44.433720 21500 webserver.cc:460] Webserver started at http://127.0.0.1:38857/ using document root <none> and password file <none>
I20260927 02:51:44.433894 21500 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:44.433933 21500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:44.434019 21500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:44.435077 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "33c26d26996945fc9230a9bd95508afc"
format_stamp: "Formatted at 2026-09-27 02:51:44 on dist-test-slave-f4rg"
server_key: "079d4bd8f23f6301e9aadfe0fba0443f"
server_key_iv: "459206afacfe28ec4676437f20f90284"
server_key_version: "encryptionkey@0"
I20260927 02:51:44.435374 21500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "33c26d26996945fc9230a9bd95508afc"
format_stamp: "Formatted at 2026-09-27 02:51:44 on dist-test-slave-f4rg"
server_key: "079d4bd8f23f6301e9aadfe0fba0443f"
server_key_iv: "459206afacfe28ec4676437f20f90284"
server_key_version: "encryptionkey@0"
I20260927 02:51:44.436527 21500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:44.437197 21515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:44.437386 21500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:44.437496 21500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "33c26d26996945fc9230a9bd95508afc"
format_stamp: "Formatted at 2026-09-27 02:51:44 on dist-test-slave-f4rg"
server_key: "079d4bd8f23f6301e9aadfe0fba0443f"
server_key_iv: "459206afacfe28ec4676437f20f90284"
server_key_version: "encryptionkey@0"
I20260927 02:51:44.437573 21500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:44.440766 21500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:44.442564 21500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:44.442672 21500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:44.446223 21500 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41805
I20260927 02:51:44.446265 21567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41805 every 8 connection(s)
I20260927 02:51:44.446559 21500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:44.447631 21568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:44.447816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21500
I20260927 02:51:44.447938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:44.448132 392 external_mini_cluster.cc:1428] Setting key 2db761f2d815492bc380f5cad18a6e15
I20260927 02:51:44.450575 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc: Bootstrap starting.
I20260927 02:51:44.450654 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:44.448665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:60511 (local address 127.0.0.1:41805)
0927 02:51:44.448822 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:44.448830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:44.448854 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:51:44.449320 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:44.449330 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:44.449364 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:44.449515 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:44.450210 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:44.450213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:44.450214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:44.450217 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:44.450274 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:44.450288 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:44.450344 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:44.450429 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 02:51:44.450473 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260927 02:51:44.451184 21568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:44.451437 21568 log.cc:824] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:44.451542 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:44.452044 21568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc: No bootstrap required, opened a new log
I20260927 02:51:44.453711 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:44.452807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:46481 (local address 127.0.0.1:41805)
0927 02:51:44.452853 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:44.452855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:44.452920 (+ 65us) server_negotiation.cc:407] Connection header received
0927 02:51:44.452981 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:44.452981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:44.452985 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:44.453027 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:44.453516 (+ 489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:44.453517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:44.453517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:44.453520 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:44.453531 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:44.453534 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:44.453571 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:44.453662 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 02:51:44.453672 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 02:51:44.454285 21568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c26d26996945fc9230a9bd95508afc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41805 } }
I20260927 02:51:44.454414 21568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:44.454434 21568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c26d26996945fc9230a9bd95508afc, State: Initialized, Role: FOLLOWER
I20260927 02:51:44.454555 21568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c26d26996945fc9230a9bd95508afc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41805 } }
I20260927 02:51:44.454627 21568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:44.454650 21568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:44.454704 21568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:44.455271 21568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c26d26996945fc9230a9bd95508afc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41805 } }
I20260927 02:51:44.455358 21568 leader_election.cc:302] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c26d26996945fc9230a9bd95508afc; no voters:
I20260927 02:51:44.455544 21568 leader_election.cc:288] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:44.455618 21578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:44.455775 21578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [term 1 LEADER]: Becoming Leader. State: Replica: 33c26d26996945fc9230a9bd95508afc, State: Running, Role: LEADER
I20260927 02:51:44.455780 21568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:44.455878 21578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c26d26996945fc9230a9bd95508afc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41805 } }
I20260927 02:51:44.456210 21580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c26d26996945fc9230a9bd95508afc. Latest consensus state: current_term: 1 leader_uuid: "33c26d26996945fc9230a9bd95508afc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c26d26996945fc9230a9bd95508afc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41805 } } }
I20260927 02:51:44.456221 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c26d26996945fc9230a9bd95508afc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c26d26996945fc9230a9bd95508afc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41805 } } }
I20260927 02:51:44.456326 21580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:44.456333 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:44.456519 21586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:44.456904 21586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:44.458501 21586 catalog_manager.cc:1347] Generated new cluster ID: fadafaa945e14a6d9b7398ddb020fdd2
I20260927 02:51:44.458560 21586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:44.463899 21586 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:44.464046 21586 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:44.467772 21586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c26d26996945fc9230a9bd95508afc: Generated new TSK 0
I20260927 02:51:44.467932 21586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:44.472602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21500
2026-09-27T02:51:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:44 dist-test-slave-f4rg krb5kdc[21599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:44 dist-test-slave-f4rg krb5kdc[21599](info): set up 2 sockets
Sep 27 02:51:44 dist-test-slave-f4rg krb5kdc[21599](info): commencing operation
krb5kdc: starting...
W20260927 02:51:45.414691 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:45 dist-test-slave-f4rg krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:45.523551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:45.600736 21612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:45.600929 21612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:45.600950 21612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:45.602059 21612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:45.602100 21612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:45.602113 21612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:45.602135 21612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:45.602152 21612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:45.603597 21612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:45.603817 21612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:45.604013 21612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:45.608387 21618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:45.608549 21612 server_base.cc:1034] running on GCE node
W20260927 02:51:45.608417 21620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:45.608449 21617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:45.608803 21612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:45.609068 21612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:45.610217 21612 hybrid_clock.cc:648] HybridClock initialized: now 1790477505610202 us; error 26 us; skew 500 ppm
Sep 27 02:51:45 dist-test-slave-f4rg krb5kdc[21599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:45.612494 21612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:45.612902 21612 webserver.cc:460] Webserver started at http://127.0.0.1:46545/ using document root <none> and password file <none>
I20260927 02:51:45.613104 21612 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:45.613150 21612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:45.613277 21612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:45.614312 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "19a135dae5b54aca9c755a5fcd41c050"
format_stamp: "Formatted at 2026-09-27 02:51:45 on dist-test-slave-f4rg"
server_key: "3a806b06284aa4aac924b3527c6374c8"
server_key_iv: "95cc6ec841430ade954d14de9bd66296"
server_key_version: "encryptionkey@0"
I20260927 02:51:45.614626 21612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "19a135dae5b54aca9c755a5fcd41c050"
format_stamp: "Formatted at 2026-09-27 02:51:45 on dist-test-slave-f4rg"
server_key: "3a806b06284aa4aac924b3527c6374c8"
server_key_iv: "95cc6ec841430ade954d14de9bd66296"
server_key_version: "encryptionkey@0"
I20260927 02:51:45.615844 21612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:45.616463 21627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:45.616612 21612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:45.616689 21612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "19a135dae5b54aca9c755a5fcd41c050"
format_stamp: "Formatted at 2026-09-27 02:51:45 on dist-test-slave-f4rg"
server_key: "3a806b06284aa4aac924b3527c6374c8"
server_key_iv: "95cc6ec841430ade954d14de9bd66296"
server_key_version: "encryptionkey@0"
I20260927 02:51:45.616755 21612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:45.633811 21612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:45.635656 21612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:45.635794 21612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:45.639452 21612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38633
I20260927 02:51:45.639492 21679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38633 every 8 connection(s)
I20260927 02:51:45.639811 21612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:45.640692 21680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:45.642939 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050: Bootstrap starting.
I20260927 02:51:45.643477 21680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:45.643734 21680 log.cc:824] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:45.644378 21680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050: No bootstrap required, opened a new log
I20260927 02:51:45.645972 21680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a135dae5b54aca9c755a5fcd41c050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 02:51:45.646109 21680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:45.646132 21680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a135dae5b54aca9c755a5fcd41c050, State: Initialized, Role: FOLLOWER
I20260927 02:51:45.646266 21680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a135dae5b54aca9c755a5fcd41c050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 02:51:45.646330 21680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:45.646351 21680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:45.646395 21680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:45.646916 21680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a135dae5b54aca9c755a5fcd41c050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 02:51:45.647001 21680 leader_election.cc:302] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19a135dae5b54aca9c755a5fcd41c050; no voters:
I20260927 02:51:45.647158 21680 leader_election.cc:288] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:45.647198 21683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:45.647329 21683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [term 1 LEADER]: Becoming Leader. State: Replica: 19a135dae5b54aca9c755a5fcd41c050, State: Running, Role: LEADER
I20260927 02:51:45.647394 21680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:45.647418 21683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a135dae5b54aca9c755a5fcd41c050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } }
I20260927 02:51:45.647792 21685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19a135dae5b54aca9c755a5fcd41c050. Latest consensus state: current_term: 1 leader_uuid: "19a135dae5b54aca9c755a5fcd41c050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a135dae5b54aca9c755a5fcd41c050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } } }
I20260927 02:51:45.647823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21612
I20260927 02:51:45.647928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:45.647930 21685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:45.647912 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19a135dae5b54aca9c755a5fcd41c050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19a135dae5b54aca9c755a5fcd41c050" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38633 } } }
I20260927 02:51:45.648030 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:45.648104 392 external_mini_cluster.cc:1428] Setting key 10aa412c02608e80e30e997856495ee2
I20260927 02:51:45.648389 21692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:45.648898 21692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:45.650465 21692 catalog_manager.cc:1347] Generated new cluster ID: 16347ae84b0145048637c9a6744ff3b8
I20260927 02:51:45.650512 21692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:51:45 dist-test-slave-f4rg krb5kdc[21599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 02:51:45.659801 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:45.648830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:35703 (local address 127.0.0.1:38633)
0927 02:51:45.649007 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:45.649017 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:45.649051 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:51:45.649245 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:45.649253 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:45.649312 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:45.649455 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:45.650179 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:45.651607 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:45.652723 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:45.653313 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:45.653398 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:45.656247 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:45.656251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:45.656251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:45.656265 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:45.658152 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:45.658552 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:45.658554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:45.658555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:45.658670 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:45.658877 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:45.658878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:45.658878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:45.659202 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:45.659364 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:45.659445 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 02:51:45.659555 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260927 02:51:45.660815 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:45.669268 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:45.662238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:47537 (local address 127.0.0.1:38633)
0927 02:51:45.662296 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:45.662298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:45.662459 (+ 161us) server_negotiation.cc:407] Connection header received
0927 02:51:45.662508 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:45.662509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:45.662525 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:45.662589 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:45.663174 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:45.663999 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:45.664955 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:45.665282 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:45.665350 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:45.666633 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:45.666634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:45.666635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:45.666642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:45.668144 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:45.668524 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:45.668525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:45.668525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:45.668585 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:45.668786 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:45.668786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:45.668787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:45.668911 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:45.669007 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:45.669109 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 02:51:45.669169 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":36}
I20260927 02:51:45.676934 21692 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:45.677461 21692 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:45.685146 21692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19a135dae5b54aca9c755a5fcd41c050: Generated new TSK 0
I20260927 02:51:45.685348 21692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:45.691519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21612
W20260927 02:51:45.695240 21597 connection.cc:535] client connection to 127.0.0.1:38633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38633): Connection reset by peer (error 104)
2026-09-27T02:51:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:45 dist-test-slave-f4rg krb5kdc[21711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:45 dist-test-slave-f4rg krb5kdc[21711](info): set up 2 sockets
Sep 27 02:51:45 dist-test-slave-f4rg krb5kdc[21711](info): commencing operation
krb5kdc: starting...
W20260927 02:51:46.638612 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:46 dist-test-slave-f4rg krb5kdc[21711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:46.758595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:46.835836 21724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:46.836031 21724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:46.836053 21724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:46.837198 21724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:46.837244 21724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:46.837256 21724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:46.837286 21724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:46.837307 21724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:46.838907 21724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:46.839138 21724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:46.839366 21724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:46.843778 21730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:46.843812 21724 server_base.cc:1034] running on GCE node
W20260927 02:51:46.843849 21729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:46.843786 21732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:46.844111 21724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:46.844370 21724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:46.845506 21724 hybrid_clock.cc:648] HybridClock initialized: now 1790477506845492 us; error 28 us; skew 500 ppm
Sep 27 02:51:46 dist-test-slave-f4rg krb5kdc[21711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:46.847793 21724 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:46.848209 21724 webserver.cc:460] Webserver started at http://127.0.0.1:43553/ using document root <none> and password file <none>
I20260927 02:51:46.848397 21724 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:46.848443 21724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:46.848562 21724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:46.849560 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "a8c0f1a76ba144868b555d4320927ba3"
format_stamp: "Formatted at 2026-09-27 02:51:46 on dist-test-slave-f4rg"
server_key: "059a58904b2eca776cf63a87fc4e1f67"
server_key_iv: "a4ca1541713bc18d63096590ae75ad8e"
server_key_version: "encryptionkey@0"
I20260927 02:51:46.849885 21724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "a8c0f1a76ba144868b555d4320927ba3"
format_stamp: "Formatted at 2026-09-27 02:51:46 on dist-test-slave-f4rg"
server_key: "059a58904b2eca776cf63a87fc4e1f67"
server_key_iv: "a4ca1541713bc18d63096590ae75ad8e"
server_key_version: "encryptionkey@0"
I20260927 02:51:46.851096 21724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:51:46.851760 21739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:46.851953 21724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:46.852041 21724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "a8c0f1a76ba144868b555d4320927ba3"
format_stamp: "Formatted at 2026-09-27 02:51:46 on dist-test-slave-f4rg"
server_key: "059a58904b2eca776cf63a87fc4e1f67"
server_key_iv: "a4ca1541713bc18d63096590ae75ad8e"
server_key_version: "encryptionkey@0"
I20260927 02:51:46.852103 21724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:46.867723 21724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:46.869403 21724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:46.869536 21724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:46.873272 21724 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36011
I20260927 02:51:46.873304 21791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36011 every 8 connection(s)
I20260927 02:51:46.873627 21724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:46.874531 21792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:46.876739 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3: Bootstrap starting.
I20260927 02:51:46.877278 21792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:46.877516 21792 log.cc:824] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:46.878120 21792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3: No bootstrap required, opened a new log
I20260927 02:51:46.879689 21792 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0f1a76ba144868b555d4320927ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36011 } }
I20260927 02:51:46.879824 21792 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:46.879845 21792 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8c0f1a76ba144868b555d4320927ba3, State: Initialized, Role: FOLLOWER
I20260927 02:51:46.879976 21792 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0f1a76ba144868b555d4320927ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36011 } }
I20260927 02:51:46.880031 21792 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:46.880061 21792 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:46.880105 21792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:46.880580 21792 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0f1a76ba144868b555d4320927ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36011 } }
I20260927 02:51:46.880666 21792 leader_election.cc:302] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8c0f1a76ba144868b555d4320927ba3; no voters:
I20260927 02:51:46.880811 21792 leader_election.cc:288] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:46.880853 21795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:46.880996 21795 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [term 1 LEADER]: Becoming Leader. State: Replica: a8c0f1a76ba144868b555d4320927ba3, State: Running, Role: LEADER
I20260927 02:51:46.881044 21792 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:46.881088 21795 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0f1a76ba144868b555d4320927ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36011 } }
I20260927 02:51:46.881484 21797 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8c0f1a76ba144868b555d4320927ba3. Latest consensus state: current_term: 1 leader_uuid: "a8c0f1a76ba144868b555d4320927ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0f1a76ba144868b555d4320927ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36011 } } }
I20260927 02:51:46.881621 21797 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:46.881594 21796 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8c0f1a76ba144868b555d4320927ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8c0f1a76ba144868b555d4320927ba3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36011 } } }
I20260927 02:51:46.881762 21796 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:46.882048 21804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:46.882505 21804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:46.882653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21724
I20260927 02:51:46.882759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:46.882953 392 external_mini_cluster.cc:1428] Setting key 2fb072ba6104e05d46dc10add664354d
I20260927 02:51:46.884034 21804 catalog_manager.cc:1347] Generated new cluster ID: 7242ec4c397c41478719db77e2052062
I20260927 02:51:46.884081 21804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:46.895864 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:46.883556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:40451 (local address 127.0.0.1:36011)
0927 02:51:46.883728 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:46.883735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:46.883759 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:51:46.883847 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:46.883852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:46.883897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:46.890963 (+ 7066us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:46.891199 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:46.892577 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:46.893693 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:46.894310 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:46.894402 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:46.895160 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:46.895165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:46.895166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:46.895379 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:46.895418 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:46.895432 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:46.895513 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:46.895605 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 02:51:46.895633 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260927 02:51:46.896427 21804 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:46.896891 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:46.897050 21804 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:46.901855 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:46.898498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:54337 (local address 127.0.0.1:36011)
0927 02:51:46.898533 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:46.898535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:46.898543 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:46.898609 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:46.898610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:46.898619 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:46.898666 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:46.898788 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:46.899598 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:46.900571 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:46.900875 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:46.900939 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:46.901609 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:46.901610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:46.901611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:46.901616 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:46.901631 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:46.901635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:46.901689 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:46.901768 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 02:51:46.901780 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:51:46.902148 21804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8c0f1a76ba144868b555d4320927ba3: Generated new TSK 0
I20260927 02:51:46.902345 21804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:46.909159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21724
W20260927 02:51:46.912822 21709 connection.cc:535] client connection to 127.0.0.1:36011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36011): Connection reset by peer (error 104)
2026-09-27T02:51:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:46 dist-test-slave-f4rg krb5kdc[21823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:46 dist-test-slave-f4rg krb5kdc[21823](info): set up 2 sockets
Sep 27 02:51:46 dist-test-slave-f4rg krb5kdc[21823](info): commencing operation
krb5kdc: starting...
W20260927 02:51:47.854626 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:47 dist-test-slave-f4rg krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 02:51:47.965209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:48.043721 21836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:48.043929 21836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:48.043951 21836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:48.045048 21836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:48.045083 21836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:48.045096 21836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:48.045107 21836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:48.045138 21836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:48.046528 21836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:48.046729 21836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:48.046924 21836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:48.051461 21842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:48.051581 21836 server_base.cc:1034] running on GCE node
W20260927 02:51:48.051465 21844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:48.051478 21841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:48.051913 21836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:48.052186 21836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:48.053328 21836 hybrid_clock.cc:648] HybridClock initialized: now 1790477508053318 us; error 31 us; skew 500 ppm
Sep 27 02:51:48 dist-test-slave-f4rg krb5kdc[21823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:48.060298 21836 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 02:51:48.060768 21836 webserver.cc:460] Webserver started at http://127.0.0.1:44797/ using document root <none> and password file <none>
I20260927 02:51:48.060951 21836 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:48.060989 21836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:48.061080 21836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:48.062084 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "043219d27ddd4cf496dc73bf7c4ee464"
format_stamp: "Formatted at 2026-09-27 02:51:48 on dist-test-slave-f4rg"
server_key: "f596706eff47d914252b488cf48e0063"
server_key_iv: "98ffc9888c36cecde52bbf82ff6c19ad"
server_key_version: "encryptionkey@0"
I20260927 02:51:48.062389 21836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "043219d27ddd4cf496dc73bf7c4ee464"
format_stamp: "Formatted at 2026-09-27 02:51:48 on dist-test-slave-f4rg"
server_key: "f596706eff47d914252b488cf48e0063"
server_key_iv: "98ffc9888c36cecde52bbf82ff6c19ad"
server_key_version: "encryptionkey@0"
I20260927 02:51:48.063612 21836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 02:51:48.064232 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:48.064375 21836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 02:51:48.064452 21836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "043219d27ddd4cf496dc73bf7c4ee464"
format_stamp: "Formatted at 2026-09-27 02:51:48 on dist-test-slave-f4rg"
server_key: "f596706eff47d914252b488cf48e0063"
server_key_iv: "98ffc9888c36cecde52bbf82ff6c19ad"
server_key_version: "encryptionkey@0"
I20260927 02:51:48.064514 21836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:48.067710 21836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:48.069674 21836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:48.069789 21836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:48.073403 21836 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44257
I20260927 02:51:48.073433 21903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44257 every 8 connection(s)
I20260927 02:51:48.073707 21836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:48.074546 21904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:48.076730 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464: Bootstrap starting.
I20260927 02:51:48.077260 21904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:48.077502 21904 log.cc:824] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:48.078114 21904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464: No bootstrap required, opened a new log
I20260927 02:51:48.079806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21836
I20260927 02:51:48.079923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:48.079880 21904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043219d27ddd4cf496dc73bf7c4ee464" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260927 02:51:48.080039 21904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:48.080070 21904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 043219d27ddd4cf496dc73bf7c4ee464, State: Initialized, Role: FOLLOWER
I20260927 02:51:48.080081 392 external_mini_cluster.cc:1428] Setting key dfbc5a44d56df33e0f0162a6dea42a49
I20260927 02:51:48.080217 21904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043219d27ddd4cf496dc73bf7c4ee464" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260927 02:51:48.080291 21904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:48.080317 21904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:48.080354 21904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:48.081205 21904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043219d27ddd4cf496dc73bf7c4ee464" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260927 02:51:48.081316 21904 leader_election.cc:302] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 043219d27ddd4cf496dc73bf7c4ee464; no voters:
I20260927 02:51:48.081486 21904 leader_election.cc:288] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:48.081514 21909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:48.081714 21909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [term 1 LEADER]: Becoming Leader. State: Replica: 043219d27ddd4cf496dc73bf7c4ee464, State: Running, Role: LEADER
I20260927 02:51:48.081808 21904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:48.081831 21909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043219d27ddd4cf496dc73bf7c4ee464" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } }
I20260927 02:51:48.082178 21909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 043219d27ddd4cf496dc73bf7c4ee464. Latest consensus state: current_term: 1 leader_uuid: "043219d27ddd4cf496dc73bf7c4ee464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043219d27ddd4cf496dc73bf7c4ee464" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260927 02:51:48.082248 21909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:48.082343 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:48.080540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:48039 (local address 127.0.0.1:44257)
0927 02:51:48.080725 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:48.080734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:48.080766 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:51:48.080870 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:48.080875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:48.080897 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:48.081009 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:48.081706 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:48.081710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:48.081710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:48.081715 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:48.081758 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:48.081773 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:48.081834 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:48.082016 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 02:51:48.082066 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260927 02:51:48.082398 21910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "043219d27ddd4cf496dc73bf7c4ee464" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "043219d27ddd4cf496dc73bf7c4ee464" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44257 } } }
I20260927 02:51:48.082522 21910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:48.083004 21914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:48.083366 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:48.083623 21914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:48.087452 21914 catalog_manager.cc:1347] Generated new cluster ID: 69a48978ea544b1cbe235ed02b8c98ac
I20260927 02:51:48.087522 21914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:48.087939 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:48.084630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.205:52735 (local address 127.0.0.1:44257)
0927 02:51:48.084666 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:48.084667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:48.087027 (+ 2360us) server_negotiation.cc:407] Connection header received
0927 02:51:48.087095 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:48.087096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:48.087101 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:48.087161 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:48.087736 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:48.087737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:48.087737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:48.087741 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:48.087757 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:48.087761 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:48.087805 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:48.087861 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:51:48.087874 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:48.099246 21914 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:48.099407 21914 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:48.111056 21914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 043219d27ddd4cf496dc73bf7c4ee464: Generated new TSK 0
I20260927 02:51:48.111279 21914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:48.116839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21836
2026-09-27T02:51:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:48 dist-test-slave-f4rg krb5kdc[21935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:48 dist-test-slave-f4rg krb5kdc[21935](info): set up 2 sockets
Sep 27 02:51:48 dist-test-slave-f4rg krb5kdc[21935](info): commencing operation
krb5kdc: starting...
W20260927 02:51:49.070736 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:49 dist-test-slave-f4rg krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:49.181804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:49.260825 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:49.261037 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:49.261070 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:49.262224 21948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:49.262267 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:49.262285 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:49.262300 21948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:49.262315 21948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:49.263841 21948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:49.264096 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:49.264304 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:49.268877 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:49.268869 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:49.268872 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:49.269217 21948 server_base.cc:1034] running on GCE node
I20260927 02:51:49.269443 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:49.269716 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:49.270861 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790477509270834 us; error 43 us; skew 500 ppm
Sep 27 02:51:49 dist-test-slave-f4rg krb5kdc[21935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:49.273191 21948 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:49.273630 21948 webserver.cc:460] Webserver started at http://127.0.98.62:43379/ using document root <none> and password file <none>
I20260927 02:51:49.273823 21948 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:49.273870 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:49.273991 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:49.275058 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "23c2806f8b284b29abe614fbb3168e2e"
format_stamp: "Formatted at 2026-09-27 02:51:49 on dist-test-slave-f4rg"
server_key: "426fc3c4777113eff253c60e601c7256"
server_key_iv: "1929f0f873b154c6c46568b2d04bf85f"
server_key_version: "encryptionkey@0"
I20260927 02:51:49.275378 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "23c2806f8b284b29abe614fbb3168e2e"
format_stamp: "Formatted at 2026-09-27 02:51:49 on dist-test-slave-f4rg"
server_key: "426fc3c4777113eff253c60e601c7256"
server_key_iv: "1929f0f873b154c6c46568b2d04bf85f"
server_key_version: "encryptionkey@0"
I20260927 02:51:49.276619 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:49.277275 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:49.277428 21948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:49.277508 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "23c2806f8b284b29abe614fbb3168e2e"
format_stamp: "Formatted at 2026-09-27 02:51:49 on dist-test-slave-f4rg"
server_key: "426fc3c4777113eff253c60e601c7256"
server_key_iv: "1929f0f873b154c6c46568b2d04bf85f"
server_key_version: "encryptionkey@0"
I20260927 02:51:49.277567 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:49.288350 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:49.290112 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:49.290249 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:49.293984 21948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44659
I20260927 02:51:49.294019 22015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44659 every 8 connection(s)
I20260927 02:51:49.294342 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:49.295198 22016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:49.296150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 21948
I20260927 02:51:49.296242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:49.296402 392 external_mini_cluster.cc:1428] Setting key 6845e9ee5d5b39c5d879ec244a36587c
I20260927 02:51:49.297905 22016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e: Bootstrap starting.
I20260927 02:51:49.298518 22016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:49.298861 22016 log.cc:824] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:49.299543 22016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e: No bootstrap required, opened a new log
I20260927 02:51:49.301673 22016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c2806f8b284b29abe614fbb3168e2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44659 } }
I20260927 02:51:49.301828 22016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:49.301851 22016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c2806f8b284b29abe614fbb3168e2e, State: Initialized, Role: FOLLOWER
I20260927 02:51:49.301962 22016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c2806f8b284b29abe614fbb3168e2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44659 } }
I20260927 02:51:49.302035 22016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:49.302063 22016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:49.302107 22016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:49.302628 22016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c2806f8b284b29abe614fbb3168e2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44659 } }
I20260927 02:51:49.302721 22016 leader_election.cc:302] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23c2806f8b284b29abe614fbb3168e2e; no voters:
I20260927 02:51:49.302906 22016 leader_election.cc:288] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:49.302951 22021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 02:51:49 dist-test-slave-f4rg krb5kdc[21935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:49.303156 22016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:49.303174 22021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [term 1 LEADER]: Becoming Leader. State: Replica: 23c2806f8b284b29abe614fbb3168e2e, State: Running, Role: LEADER
I20260927 02:51:49.303299 22021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c2806f8b284b29abe614fbb3168e2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44659 } }
I20260927 02:51:49.303697 22023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23c2806f8b284b29abe614fbb3168e2e. Latest consensus state: current_term: 1 leader_uuid: "23c2806f8b284b29abe614fbb3168e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c2806f8b284b29abe614fbb3168e2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44659 } } }
I20260927 02:51:49.303712 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23c2806f8b284b29abe614fbb3168e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23c2806f8b284b29abe614fbb3168e2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44659 } } }
I20260927 02:51:49.303791 22023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:49.303813 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:49.304021 22030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:49.304546 22030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:49.307310 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:49.296876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34729 (local address 127.0.98.62:44659)
0927 02:51:49.297058 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:49.297067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:49.297098 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:51:49.297580 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:49.297589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:49.297652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:49.297858 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:49.298422 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:49.299727 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:49.300446 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:49.301052 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:49.301190 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:49.303826 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:49.303831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:49.303831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:49.303844 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:49.305723 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:49.306154 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:49.306156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:49.306157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:49.306237 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:49.306468 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:49.306469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:49.306469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:49.306719 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:49.306854 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:49.307028 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 02:51:49.307129 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260927 02:51:49.307928 22030 catalog_manager.cc:1347] Generated new cluster ID: 6a2f8f78452249d4837670090307566d
I20260927 02:51:49.307979 22030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:49.308346 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:49.314966 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:49.309505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49299 (local address 127.0.98.62:44659)
0927 02:51:49.309537 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:49.309539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:49.309644 (+ 105us) server_negotiation.cc:407] Connection header received
0927 02:51:49.309696 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:49.309697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:49.309706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:49.309750 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:49.310208 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:49.310673 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:49.311399 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:49.311628 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:49.311682 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:49.312750 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:49.312751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:49.312752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:49.312758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:49.313952 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:49.314280 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:49.314281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:49.314281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:49.314331 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:49.314510 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:49.314511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:49.314511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:49.314610 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:49.314715 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:49.314832 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 02:51:49.314884 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 02:51:49.325824 22030 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:49.327145 22030 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:49.340083 22030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23c2806f8b284b29abe614fbb3168e2e: Generated new TSK 0
I20260927 02:51:49.340301 22030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:49.344674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 21948
W20260927 02:51:49.348573 21933 connection.cc:535] client connection to 127.0.98.62:44659 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44659): Connection reset by peer (error 104)
2026-09-27T02:51:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:49 dist-test-slave-f4rg krb5kdc[22047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:49 dist-test-slave-f4rg krb5kdc[22047](info): set up 2 sockets
Sep 27 02:51:49 dist-test-slave-f4rg krb5kdc[22047](info): commencing operation
krb5kdc: starting...
W20260927 02:51:50.294638 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:50 dist-test-slave-f4rg krb5kdc[22047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:50.405505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:50.482995 22060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:50.483182 22060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:50.483203 22060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:50.484330 22060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:50.484371 22060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:50.484385 22060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:50.484395 22060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:50.484405 22060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:50.485816 22060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:50.486033 22060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:50.486227 22060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:51:50.490847 22060 server_base.cc:1034] running on GCE node
W20260927 02:51:50.490777 22065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:50.490821 22066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:50.490779 22068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:50.491269 22060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:50.491540 22060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:50.492693 22060 hybrid_clock.cc:648] HybridClock initialized: now 1790477510492683 us; error 26 us; skew 500 ppm
Sep 27 02:51:50 dist-test-slave-f4rg krb5kdc[22047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:50.494997 22060 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:50.495380 22060 webserver.cc:460] Webserver started at http://127.0.98.62:35081/ using document root <none> and password file <none>
I20260927 02:51:50.495575 22060 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:50.495620 22060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:50.495782 22060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:50.496758 22060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "631a1e6db04f4c29802c37f555d91622"
format_stamp: "Formatted at 2026-09-27 02:51:50 on dist-test-slave-f4rg"
server_key: "5c3144ddf673dd2f6630747840afaebc"
server_key_iv: "eae646370e1b97252b407e7b4490ad7b"
server_key_version: "encryptionkey@0"
I20260927 02:51:50.497068 22060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "631a1e6db04f4c29802c37f555d91622"
format_stamp: "Formatted at 2026-09-27 02:51:50 on dist-test-slave-f4rg"
server_key: "5c3144ddf673dd2f6630747840afaebc"
server_key_iv: "eae646370e1b97252b407e7b4490ad7b"
server_key_version: "encryptionkey@0"
I20260927 02:51:50.498215 22060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:51:50.498842 22075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:50.498974 22060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:50.499045 22060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "631a1e6db04f4c29802c37f555d91622"
format_stamp: "Formatted at 2026-09-27 02:51:50 on dist-test-slave-f4rg"
server_key: "5c3144ddf673dd2f6630747840afaebc"
server_key_iv: "eae646370e1b97252b407e7b4490ad7b"
server_key_version: "encryptionkey@0"
I20260927 02:51:50.499109 22060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:50.508836 22060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:50.510784 22060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:50.510928 22060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:50.514528 22060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32805
I20260927 02:51:50.514590 22127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32805 every 8 connection(s)
I20260927 02:51:50.514918 22060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:50.515715 22128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:50.517835 22128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622: Bootstrap starting.
I20260927 02:51:50.518363 22128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:50.518603 22128 log.cc:824] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:50.519251 22128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622: No bootstrap required, opened a new log
I20260927 02:51:50.519711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22060
I20260927 02:51:50.519835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:50.520007 392 external_mini_cluster.cc:1428] Setting key 761b6ef7dc59f7054c1a5e526a858496
I20260927 02:51:50.521183 22128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a1e6db04f4c29802c37f555d91622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32805 } }
I20260927 02:51:50.521322 22128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:50.521354 22128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631a1e6db04f4c29802c37f555d91622, State: Initialized, Role: FOLLOWER
I20260927 02:51:50.521535 22128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a1e6db04f4c29802c37f555d91622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32805 } }
I20260927 02:51:50.521620 22128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:50.521653 22128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:50.521698 22128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:50.522181 22128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a1e6db04f4c29802c37f555d91622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32805 } }
I20260927 02:51:50.522267 22128 leader_election.cc:302] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 631a1e6db04f4c29802c37f555d91622; no voters:
I20260927 02:51:50.522410 22128 leader_election.cc:288] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:50.522439 22133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:50.522599 22133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [term 1 LEADER]: Becoming Leader. State: Replica: 631a1e6db04f4c29802c37f555d91622, State: Running, Role: LEADER
I20260927 02:51:50.522641 22128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:50.522764 22133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a1e6db04f4c29802c37f555d91622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32805 } }
I20260927 02:51:50.523285 22134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "631a1e6db04f4c29802c37f555d91622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a1e6db04f4c29802c37f555d91622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32805 } } }
I20260927 02:51:50.523357 22134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:50.523569 22135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 631a1e6db04f4c29802c37f555d91622. Latest consensus state: current_term: 1 leader_uuid: "631a1e6db04f4c29802c37f555d91622" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "631a1e6db04f4c29802c37f555d91622" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32805 } } }
I20260927 02:51:50.523665 22135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:50.524160 22142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:50.524601 22142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:50.526012 22142 catalog_manager.cc:1347] Generated new cluster ID: 5d38301fcb694f3d9370247dc5ce099e
I20260927 02:51:50.526060 22142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:50.526574 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:50.520504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58101 (local address 127.0.98.62:32805)
0927 02:51:50.520675 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:50.520685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:50.520716 (+ 31us) server_negotiation.cc:407] Connection header received
0927 02:51:50.521160 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:50.521166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:50.521209 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:50.521315 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:50.521572 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:50.523044 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:50.524099 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:50.524678 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:50.524780 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:50.525876 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:50.525879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:50.525880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:50.526090 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:50.526128 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:50.526143 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:50.526239 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:50.526334 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 02:51:50.526364 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260927 02:51:50.527448 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:50.536623 22142 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:50.537302 22142 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:50.538367 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:50.528795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49651 (local address 127.0.98.62:32805)
0927 02:51:50.528851 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:50.528853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:50.534985 (+ 6132us) server_negotiation.cc:407] Connection header received
0927 02:51:50.535045 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:50.535046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:50.535072 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:50.535139 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:50.535275 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:50.536058 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:50.536986 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:50.537305 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:50.537346 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:50.538120 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:50.538121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:50.538122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:50.538127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:50.538145 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:50.538150 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:50.538207 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:50.538272 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 02:51:50.538285 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":37}
I20260927 02:51:50.546484 22142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 631a1e6db04f4c29802c37f555d91622: Generated new TSK 0
I20260927 02:51:50.546675 22142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:50.553028 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22060
W20260927 02:51:50.556608 22045 connection.cc:535] client connection to 127.0.98.62:32805 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:32805): Connection reset by peer (error 104)
2026-09-27T02:51:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:50 dist-test-slave-f4rg krb5kdc[22159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:50 dist-test-slave-f4rg krb5kdc[22159](info): set up 2 sockets
Sep 27 02:51:50 dist-test-slave-f4rg krb5kdc[22159](info): commencing operation
krb5kdc: starting...
W20260927 02:51:51.498543 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:51 dist-test-slave-f4rg krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:51.603351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33557
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:51.680583 22172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:51.680778 22172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:51.680809 22172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:51.681910 22172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:51.681950 22172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:51.681969 22172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:51.681984 22172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:51.681999 22172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:51.683511 22172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:51.683753 22172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:51.683943 22172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:51.688388 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:51.688426 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:51.688546 22172 server_base.cc:1034] running on GCE node
W20260927 02:51:51.688413 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:51.688868 22172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:51.689132 22172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:51.690271 22172 hybrid_clock.cc:648] HybridClock initialized: now 1790477511690254 us; error 29 us; skew 500 ppm
Sep 27 02:51:51 dist-test-slave-f4rg krb5kdc[22159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:51.692523 22172 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:51.692919 22172 webserver.cc:460] Webserver started at http://127.0.98.62:36111/ using document root <none> and password file <none>
I20260927 02:51:51.693109 22172 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:51.693151 22172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:51.693267 22172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:51.694262 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "a05c609462544ef0a4b25739ef5bdbe5"
format_stamp: "Formatted at 2026-09-27 02:51:51 on dist-test-slave-f4rg"
server_key: "9d711031a90f1c793e12d746d17321be"
server_key_iv: "8e81c624d93684110472bdfe3678a617"
server_key_version: "encryptionkey@0"
I20260927 02:51:51.694564 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "a05c609462544ef0a4b25739ef5bdbe5"
format_stamp: "Formatted at 2026-09-27 02:51:51 on dist-test-slave-f4rg"
server_key: "9d711031a90f1c793e12d746d17321be"
server_key_iv: "8e81c624d93684110472bdfe3678a617"
server_key_version: "encryptionkey@0"
I20260927 02:51:51.695818 22172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 02:51:51.696456 22187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:51.696589 22172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:51.696662 22172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "a05c609462544ef0a4b25739ef5bdbe5"
format_stamp: "Formatted at 2026-09-27 02:51:51 on dist-test-slave-f4rg"
server_key: "9d711031a90f1c793e12d746d17321be"
server_key_iv: "8e81c624d93684110472bdfe3678a617"
server_key_version: "encryptionkey@0"
I20260927 02:51:51.696732 22172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:51.699834 22172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:51.701464 22172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:51.701576 22172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:51.705010 22172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33557
I20260927 02:51:51.705049 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33557 every 8 connection(s)
I20260927 02:51:51.705307 22172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:51.706162 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:51.707015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22172
I20260927 02:51:51.707127 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:51.707310 392 external_mini_cluster.cc:1428] Setting key b75b3a1b832536531438fd6cfb590b94
I20260927 02:51:51.708684 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5: Bootstrap starting.
I20260927 02:51:51.709184 22240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:51.709400 22240 log.cc:824] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:51.709982 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5: No bootstrap required, opened a new log
I20260927 02:51:51.711606 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:51.707795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50979 (local address 127.0.98.62:33557)
0927 02:51:51.707956 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:51.707965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:51.709713 (+ 1748us) server_negotiation.cc:407] Connection header received
0927 02:51:51.710187 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:51.710194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:51.710220 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:51.710348 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:51.711026 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:51.711029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:51.711029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:51.711033 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:51.711072 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:51.711082 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:51.711151 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:51.711211 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:51:51.711252 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260927 02:51:51.711521 22240 raft_consensus.cc:348] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c609462544ef0a4b25739ef5bdbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260927 02:51:51.711668 22240 raft_consensus.cc:374] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:51.711704 22240 raft_consensus.cc:729] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05c609462544ef0a4b25739ef5bdbe5, State: Initialized, Role: FOLLOWER
I20260927 02:51:51.711835 22240 consensus_queue.cc:260] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c609462544ef0a4b25739ef5bdbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260927 02:51:51.711927 22240 raft_consensus.cc:388] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:51.711966 22240 raft_consensus.cc:482] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:51.712010 22240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:51.712505 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:51.712599 22240 raft_consensus.cc:504] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c609462544ef0a4b25739ef5bdbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260927 02:51:51.712702 22240 leader_election.cc:302] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a05c609462544ef0a4b25739ef5bdbe5; no voters:
I20260927 02:51:51.712872 22240 leader_election.cc:288] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:51.712917 22245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:51.713050 22245 raft_consensus.cc:686] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [term 1 LEADER]: Becoming Leader. State: Replica: a05c609462544ef0a4b25739ef5bdbe5, State: Running, Role: LEADER
I20260927 02:51:51.713093 22240 sys_catalog.cc:564] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:51.713169 22245 consensus_queue.cc:237] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c609462544ef0a4b25739ef5bdbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } }
I20260927 02:51:51.713495 22245 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a05c609462544ef0a4b25739ef5bdbe5. Latest consensus state: current_term: 1 leader_uuid: "a05c609462544ef0a4b25739ef5bdbe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c609462544ef0a4b25739ef5bdbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } } }
I20260927 02:51:51.713567 22245 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:51.713547 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a05c609462544ef0a4b25739ef5bdbe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05c609462544ef0a4b25739ef5bdbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33557 } } }
I20260927 02:51:51.713694 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:51.713893 22258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:51.714403 22258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:51.715806 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:51.713951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47875 (local address 127.0.98.62:33557)
0927 02:51:51.714128 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:51.714130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:51.714999 (+ 869us) server_negotiation.cc:407] Connection header received
0927 02:51:51.715022 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:51.715023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:51.715027 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:51.715078 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:51.715608 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:51.715609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:51.715609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:51.715613 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:51.715625 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:51.715635 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:51.715680 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:51.715736 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:51:51.715749 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260927 02:51:51.719723 22258 catalog_manager.cc:1347] Generated new cluster ID: b3ac46cfa5644a2e9e42d36b865729a9
I20260927 02:51:51.719774 22258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:51.756353 22258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:51.756523 22258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:51.761006 22258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a05c609462544ef0a4b25739ef5bdbe5: Generated new TSK 0
I20260927 02:51:51.761201 22258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:51.768370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22172
2026-09-27T02:51:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:51 dist-test-slave-f4rg krb5kdc[22271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:51 dist-test-slave-f4rg krb5kdc[22271](info): set up 2 sockets
Sep 27 02:51:51 dist-test-slave-f4rg krb5kdc[22271](info): commencing operation
krb5kdc: starting...
W20260927 02:51:52.714702 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:52 dist-test-slave-f4rg krb5kdc[22271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:52.823372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:52.900700 22284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:52.900895 22284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:52.900916 22284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:52.901978 22284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:52.902014 22284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:52.902025 22284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:52.902036 22284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:52.902052 22284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:52.903438 22284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:52.903617 22284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:52.903787 22284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:52.908314 22290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:52.908406 22292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:52.908389 22284 server_base.cc:1034] running on GCE node
W20260927 02:51:52.908375 22289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:52.908804 22284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:52.909071 22284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:52.910207 22284 hybrid_clock.cc:648] HybridClock initialized: now 1790477512910195 us; error 26 us; skew 500 ppm
Sep 27 02:51:52 dist-test-slave-f4rg krb5kdc[22271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:52.912519 22284 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:52.912935 22284 webserver.cc:460] Webserver started at http://127.0.98.62:44171/ using document root <none> and password file <none>
I20260927 02:51:52.913132 22284 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:52.913175 22284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:52.913285 22284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:52.914296 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "ec785a29644842a2b737ada1190b838c"
format_stamp: "Formatted at 2026-09-27 02:51:52 on dist-test-slave-f4rg"
server_key: "d11d6f25d6c98f683d0a17d65b8cc211"
server_key_iv: "513777488db9ba718c192682d3ab2909"
server_key_version: "encryptionkey@0"
I20260927 02:51:52.914602 22284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "ec785a29644842a2b737ada1190b838c"
format_stamp: "Formatted at 2026-09-27 02:51:52 on dist-test-slave-f4rg"
server_key: "d11d6f25d6c98f683d0a17d65b8cc211"
server_key_iv: "513777488db9ba718c192682d3ab2909"
server_key_version: "encryptionkey@0"
I20260927 02:51:52.915866 22284 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:52.916584 22299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:52.916767 22284 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:52.916841 22284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "ec785a29644842a2b737ada1190b838c"
format_stamp: "Formatted at 2026-09-27 02:51:52 on dist-test-slave-f4rg"
server_key: "d11d6f25d6c98f683d0a17d65b8cc211"
server_key_iv: "513777488db9ba718c192682d3ab2909"
server_key_version: "encryptionkey@0"
I20260927 02:51:52.916895 22284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:52.929100 22284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:52.930997 22284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:52.931105 22284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:52.934500 22284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44747
I20260927 02:51:52.934546 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44747 every 8 connection(s)
I20260927 02:51:52.934876 22284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:52.935655 22352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:52.937616 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22284
I20260927 02:51:52.937737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:52.937878 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c: Bootstrap starting.
I20260927 02:51:52.937927 392 external_mini_cluster.cc:1428] Setting key fb37450ffce3a54217203dfc71a6e83b
I20260927 02:51:52.938519 22352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:52.938851 22352 log.cc:824] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:52.939553 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c: No bootstrap required, opened a new log
I20260927 02:51:52.941736 22352 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec785a29644842a2b737ada1190b838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44747 } }
I20260927 02:51:52.941867 22352 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:52.941897 22352 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec785a29644842a2b737ada1190b838c, State: Initialized, Role: FOLLOWER
I20260927 02:51:52.942018 22352 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec785a29644842a2b737ada1190b838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44747 } }
I20260927 02:51:52.942098 22352 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:52.942137 22352 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:52.942183 22352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:52.942775 22352 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec785a29644842a2b737ada1190b838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44747 } }
I20260927 02:51:52.942900 22352 leader_election.cc:302] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec785a29644842a2b737ada1190b838c; no voters:
I20260927 02:51:52.943051 22352 leader_election.cc:288] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:52.943089 22357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:52.943234 22357 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [term 1 LEADER]: Becoming Leader. State: Replica: ec785a29644842a2b737ada1190b838c, State: Running, Role: LEADER
I20260927 02:51:52.943303 22352 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:52.943358 22357 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec785a29644842a2b737ada1190b838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44747 } }
I20260927 02:51:52.944577 22359 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec785a29644842a2b737ada1190b838c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec785a29644842a2b737ada1190b838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44747 } } }
I20260927 02:51:52.944717 22359 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [sys.catalog]: This master's current role is: LEADER
Sep 27 02:51:52 dist-test-slave-f4rg krb5kdc[22271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:52.944770 22358 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec785a29644842a2b737ada1190b838c. Latest consensus state: current_term: 1 leader_uuid: "ec785a29644842a2b737ada1190b838c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec785a29644842a2b737ada1190b838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44747 } } }
I20260927 02:51:52.945967 22358 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c [sys.catalog]: This master's current role is: LEADER
W20260927 02:51:52.945170 22371 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:51:52.946229 22371 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:52.946492 22372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:52.946693 22372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:52.948385 22372 catalog_manager.cc:1347] Generated new cluster ID: f830c457156449a183a27f12889043a5
I20260927 02:51:52.948433 22372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:52.949237 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:52.938423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42419 (local address 127.0.98.62:44747)
0927 02:51:52.938623 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:52.938634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:52.938668 (+ 34us) server_negotiation.cc:407] Connection header received
0927 02:51:52.939232 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:52.939240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:52.939303 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:52.939439 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:52.940028 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:52.941434 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:52.942450 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:52.943082 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:52.943512 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:52.945691 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:52.945694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:52.945694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:52.945704 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:52.947661 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:52.947998 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:52.948000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:52.948001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:52.948107 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:52.948329 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:52.948330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:52.948331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:52.948637 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:52.948792 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:52.948897 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 02:51:52.949032 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":57,"threads_started":1}
I20260927 02:51:52.950242 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:52.958250 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:52.951632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60297 (local address 127.0.98.62:44747)
0927 02:51:52.951662 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:52.951664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:52.951780 (+ 116us) server_negotiation.cc:407] Connection header received
0927 02:51:52.951839 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:52.951839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:52.951851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:52.951902 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:52.952504 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:52.953207 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:52.954172 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:52.954462 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:52.954512 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:52.955832 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:52.955833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:52.955833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:52.955839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:52.957254 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:52.957610 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:52.957611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:52.957611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:52.957657 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:52.957822 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:52.957822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:52.957823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:52.957925 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:52.957999 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:52.958099 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 02:51:52.958166 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:51:52.968659 22372 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:52.969177 22372 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:52.971920 22372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec785a29644842a2b737ada1190b838c: Generated new TSK 0
I20260927 02:51:52.972095 22372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:52.981696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22284
2026-09-27T02:51:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:53 dist-test-slave-f4rg krb5kdc[22383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:53 dist-test-slave-f4rg krb5kdc[22383](info): set up 2 sockets
Sep 27 02:51:53 dist-test-slave-f4rg krb5kdc[22383](info): commencing operation
krb5kdc: starting...
W20260927 02:51:53.938617 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:54 dist-test-slave-f4rg krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:54.047844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40137
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:54.125665 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:54.125880 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:54.125902 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:54.127046 22396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:54.127082 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:54.127095 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:54.127106 22396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:54.127116 22396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:54.128580 22396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:54.128783 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:54.128960 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:54.133773 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:54.133723 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:54.133776 22396 server_base.cc:1034] running on GCE node
W20260927 02:51:54.133744 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:54.134187 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:54.134485 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:54.135623 22396 hybrid_clock.cc:648] HybridClock initialized: now 1790477514135610 us; error 26 us; skew 500 ppm
Sep 27 02:51:54 dist-test-slave-f4rg krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:54.142766 22396 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:54.143226 22396 webserver.cc:460] Webserver started at http://127.0.98.62:39551/ using document root <none> and password file <none>
I20260927 02:51:54.143419 22396 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:54.143462 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:54.143558 22396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:54.144596 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "4ebcd4a32a4c429f94ceddb3607e69ff"
format_stamp: "Formatted at 2026-09-27 02:51:54 on dist-test-slave-f4rg"
server_key: "45842e9eeb6f142f1dbd19c4aab44b5f"
server_key_iv: "f30d9b407f9f31f563f500bc66a4f0e8"
server_key_version: "encryptionkey@0"
I20260927 02:51:54.144901 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ebcd4a32a4c429f94ceddb3607e69ff"
format_stamp: "Formatted at 2026-09-27 02:51:54 on dist-test-slave-f4rg"
server_key: "45842e9eeb6f142f1dbd19c4aab44b5f"
server_key_iv: "f30d9b407f9f31f563f500bc66a4f0e8"
server_key_version: "encryptionkey@0"
I20260927 02:51:54.146111 22396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 02:51:54.146850 22411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:54.147023 22396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 02:51:54.147111 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "4ebcd4a32a4c429f94ceddb3607e69ff"
format_stamp: "Formatted at 2026-09-27 02:51:54 on dist-test-slave-f4rg"
server_key: "45842e9eeb6f142f1dbd19c4aab44b5f"
server_key_iv: "f30d9b407f9f31f563f500bc66a4f0e8"
server_key_version: "encryptionkey@0"
I20260927 02:51:54.147171 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:54.165750 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:54.167644 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:54.167760 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:54.171353 22396 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40137
I20260927 02:51:54.171399 22463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40137 every 8 connection(s)
I20260927 02:51:54.171710 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:54.172152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22396
I20260927 02:51:54.172240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:54.172395 392 external_mini_cluster.cc:1428] Setting key 6fae04b4c1453e05379733ee809e6175
I20260927 02:51:54.172609 22464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:54.176039 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff: Bootstrap starting.
I20260927 02:51:54.176649 22464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:54.176926 22464 log.cc:824] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:54.177634 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff: No bootstrap required, opened a new log
I20260927 02:51:54.178555 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:54.172896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51583 (local address 127.0.98.62:40137)
0927 02:51:54.173109 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:54.173118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:54.173150 (+ 32us) server_negotiation.cc:407] Connection header received
0927 02:51:54.173855 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:54.173866 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:54.173934 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:54.174090 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:54.174309 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:54.175680 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:54.176517 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:54.177116 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:54.177208 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:54.177888 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:54.177892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:54.177893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:54.178089 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:54.178129 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:54.178143 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:54.178209 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:54.178268 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 02:51:54.178334 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
I20260927 02:51:54.179525 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:54.179833 22464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40137 } }
I20260927 02:51:54.179965 22464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:54.179987 22464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebcd4a32a4c429f94ceddb3607e69ff, State: Initialized, Role: FOLLOWER
I20260927 02:51:54.180109 22464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40137 } }
I20260927 02:51:54.180171 22464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:54.180190 22464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:54.180235 22464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:54.180898 22464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40137 } }
I20260927 02:51:54.181011 22464 leader_election.cc:302] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ebcd4a32a4c429f94ceddb3607e69ff; no voters:
I20260927 02:51:54.181183 22464 leader_election.cc:288] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:54.181222 22474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:54.181383 22474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [term 1 LEADER]: Becoming Leader. State: Replica: 4ebcd4a32a4c429f94ceddb3607e69ff, State: Running, Role: LEADER
I20260927 02:51:54.181437 22464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:54.181542 22474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40137 } }
I20260927 02:51:54.181903 22475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40137 } } }
I20260927 02:51:54.181917 22476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ebcd4a32a4c429f94ceddb3607e69ff. Latest consensus state: current_term: 1 leader_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ebcd4a32a4c429f94ceddb3607e69ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40137 } } }
I20260927 02:51:54.181995 22475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:54.182006 22476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:54.182317 22483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:54.182884 22483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:54.183882 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:54.180812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46533 (local address 127.0.98.62:40137)
0927 02:51:54.180853 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:54.180854 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:54.180950 (+ 96us) server_negotiation.cc:407] Connection header received
0927 02:51:54.181010 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:54.181010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:54.181020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:54.181074 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:54.181214 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:54.181880 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:54.182620 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:54.182935 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:54.182996 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:54.183664 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:54.183665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:54.183665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:54.183669 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:54.183683 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:54.183687 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:54.183742 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:54.183796 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 02:51:54.183816 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:51:54.184763 22483 catalog_manager.cc:1347] Generated new cluster ID: 6e346f14db82444093a9e286473731c7
I20260927 02:51:54.184803 22483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:54.187734 22483 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:54.188287 22483 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:54.193495 22483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ebcd4a32a4c429f94ceddb3607e69ff: Generated new TSK 0
I20260927 02:51:54.193674 22483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:54.196230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22396
2026-09-27T02:51:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:54 dist-test-slave-f4rg krb5kdc[22495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:54 dist-test-slave-f4rg krb5kdc[22495](info): set up 2 sockets
Sep 27 02:51:54 dist-test-slave-f4rg krb5kdc[22495](info): commencing operation
krb5kdc: starting...
W20260927 02:51:55.134692 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:55 dist-test-slave-f4rg krb5kdc[22495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:55.245995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:55.324595 22508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:55.324805 22508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:55.324833 22508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:55.325953 22508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:55.325995 22508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:55.326022 22508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:55.326040 22508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:55.326062 22508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:55.327564 22508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:55.327800 22508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:55.327997 22508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:55.333065 22513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:55.333056 22514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:55.333057 22516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:55.333240 22508 server_base.cc:1034] running on GCE node
I20260927 02:51:55.333590 22508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:55.333873 22508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:55.335013 22508 hybrid_clock.cc:648] HybridClock initialized: now 1790477515334998 us; error 32 us; skew 500 ppm
Sep 27 02:51:55 dist-test-slave-f4rg krb5kdc[22495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:55.337296 22508 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:55.337725 22508 webserver.cc:460] Webserver started at http://127.0.98.62:33985/ using document root <none> and password file <none>
I20260927 02:51:55.337913 22508 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:55.337960 22508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:55.338080 22508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:55.339138 22508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "afe0ddd8bc594cbda70c888f14dae732"
format_stamp: "Formatted at 2026-09-27 02:51:55 on dist-test-slave-f4rg"
server_key: "b57bf2f8c0ba6ea45d44b3fa0084f166"
server_key_iv: "7af203a94eb9391029883714d8699df7"
server_key_version: "encryptionkey@0"
I20260927 02:51:55.339453 22508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "afe0ddd8bc594cbda70c888f14dae732"
format_stamp: "Formatted at 2026-09-27 02:51:55 on dist-test-slave-f4rg"
server_key: "b57bf2f8c0ba6ea45d44b3fa0084f166"
server_key_iv: "7af203a94eb9391029883714d8699df7"
server_key_version: "encryptionkey@0"
I20260927 02:51:55.340664 22508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:55.341311 22523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:55.341437 22508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:55.341514 22508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "afe0ddd8bc594cbda70c888f14dae732"
format_stamp: "Formatted at 2026-09-27 02:51:55 on dist-test-slave-f4rg"
server_key: "b57bf2f8c0ba6ea45d44b3fa0084f166"
server_key_iv: "7af203a94eb9391029883714d8699df7"
server_key_version: "encryptionkey@0"
I20260927 02:51:55.341574 22508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:55.344738 22508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:55.346382 22508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:55.346500 22508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:55.350011 22508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42065
I20260927 02:51:55.350039 22575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42065 every 8 connection(s)
I20260927 02:51:55.350342 22508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:55.351255 22576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:55.353349 22576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732: Bootstrap starting.
I20260927 02:51:55.353852 22576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:55.354089 22576 log.cc:824] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:55.354668 22576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732: No bootstrap required, opened a new log
I20260927 02:51:55.356195 22576 raft_consensus.cc:348] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe0ddd8bc594cbda70c888f14dae732" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42065 } }
I20260927 02:51:55.356345 22576 raft_consensus.cc:374] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 02:51:55.356371 22576 raft_consensus.cc:729] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe0ddd8bc594cbda70c888f14dae732, State: Initialized, Role: FOLLOWER
I20260927 02:51:55.356508 22576 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe0ddd8bc594cbda70c888f14dae732" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42065 } }
I20260927 02:51:55.356578 22576 raft_consensus.cc:388] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 02:51:55.356611 22576 raft_consensus.cc:482] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 02:51:55.356657 22576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:55.357122 22576 raft_consensus.cc:504] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe0ddd8bc594cbda70c888f14dae732" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42065 } }
I20260927 02:51:55.357206 22576 leader_election.cc:302] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe0ddd8bc594cbda70c888f14dae732; no voters:
I20260927 02:51:55.357363 22576 leader_election.cc:288] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 02:51:55.357402 22579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:55.357559 22579 raft_consensus.cc:686] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [term 1 LEADER]: Becoming Leader. State: Replica: afe0ddd8bc594cbda70c888f14dae732, State: Running, Role: LEADER
I20260927 02:51:55.357587 22576 sys_catalog.cc:564] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:55.357645 22579 consensus_queue.cc:237] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe0ddd8bc594cbda70c888f14dae732" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42065 } }
I20260927 02:51:55.357968 22581 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe0ddd8bc594cbda70c888f14dae732. Latest consensus state: current_term: 1 leader_uuid: "afe0ddd8bc594cbda70c888f14dae732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe0ddd8bc594cbda70c888f14dae732" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42065 } } }
I20260927 02:51:55.358075 22581 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:55.357959 22580 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afe0ddd8bc594cbda70c888f14dae732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe0ddd8bc594cbda70c888f14dae732" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42065 } } }
I20260927 02:51:55.358186 22580 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:55.358259 22583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:55.358796 22583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:55.360188 22583 catalog_manager.cc:1347] Generated new cluster ID: 74f968bdc42a4e9fbce4859f0142decc
I20260927 02:51:55.360239 22583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:55.360387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22508
I20260927 02:51:55.360502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:55.360666 392 external_mini_cluster.cc:1428] Setting key 9f51d8d2ea90448e776e99d02aaedb4c
I20260927 02:51:55.364493 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:55.361080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50063 (local address 127.0.98.62:42065)
0927 02:51:55.361244 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:55.361254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:55.363018 (+ 1764us) server_negotiation.cc:407] Connection header received
0927 02:51:55.363169 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:55.363176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:55.363205 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:55.363342 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:55.364087 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:55.364090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:55.364091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:55.364094 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:55.364132 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:55.364145 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:55.364207 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:55.364285 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 02:51:55.364321 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260927 02:51:55.365291 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:55.367695 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:55.366645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39471 (local address 127.0.98.62:42065)
0927 02:51:55.366674 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:55.366675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:55.366784 (+ 109us) server_negotiation.cc:407] Connection header received
0927 02:51:55.366875 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:55.366875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:55.366892 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:55.366939 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:55.367473 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:55.367474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:55.367475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 02:51:55.367478 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:55.367491 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 02:51:55.367495 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:55.367559 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:55.367615 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 02:51:55.367627 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 02:51:55.382155 22583 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:55.382349 22583 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:55.387727 22583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afe0ddd8bc594cbda70c888f14dae732: Generated new TSK 0
I20260927 02:51:55.387921 22583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:55.396219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22508
2026-09-27T02:51:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1200 ms)
[----------] 18 tests from AuthTokenIssuingTest (21798 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:55 dist-test-slave-f4rg krb5kdc[22607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:55 dist-test-slave-f4rg krb5kdc[22607](info): set up 2 sockets
Sep 27 02:51:55 dist-test-slave-f4rg krb5kdc[22607](info): commencing operation
krb5kdc: starting...
W20260927 02:51:56.338706 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:56.447541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34553
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34553,127.0.98.61:45029,127.0.98.60:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:56.525161 22620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:51:56.525372 22620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:56.525396 22620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:56.525419 22620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:56.526552 22620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:56.526594 22620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:56.526612 22620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:56.526628 22620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:56.526643 22620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:56.528160 22620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34553,127.0.98.61:45029,127.0.98.60:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:56.528406 22620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:56.528611 22620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:56.533392 22628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:56.533412 22626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:56.533407 22625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:56.533485 22620 server_base.cc:1034] running on GCE node
I20260927 02:51:56.533879 22620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:56.534178 22620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:56.535349 22620 hybrid_clock.cc:648] HybridClock initialized: now 1790477516535337 us; error 27 us; skew 500 ppm
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:56.537763 22620 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:56.538164 22620 webserver.cc:460] Webserver started at http://127.0.98.62:33999/ using document root <none> and password file <none>
I20260927 02:51:56.538367 22620 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:56.538409 22620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:56.538535 22620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:56.539611 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "01a323ff4756466ba748bb7eb9fd8914"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "658a23c804f9232e8393f569bb15aea1"
server_key_iv: "ec86ee7e69fe02a44e0bf11957f6af35"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.539935 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "01a323ff4756466ba748bb7eb9fd8914"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "658a23c804f9232e8393f569bb15aea1"
server_key_iv: "ec86ee7e69fe02a44e0bf11957f6af35"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.541112 22620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:56.541796 22635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:56.541934 22620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:56.542001 22620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "01a323ff4756466ba748bb7eb9fd8914"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "658a23c804f9232e8393f569bb15aea1"
server_key_iv: "ec86ee7e69fe02a44e0bf11957f6af35"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.542064 22620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:56.550781 22620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:56.552794 22620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:56.552915 22620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:56.556453 22620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34553
I20260927 02:51:56.556509 22687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34553 every 8 connection(s)
I20260927 02:51:56.556809 22620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:56.557713 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:56.559204 22688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:56.561772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22620
I20260927 02:51:56.561867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:56.562036 392 external_mini_cluster.cc:1428] Setting key 4fa009e22ed30904a9b9df43913f848b
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:56.568192 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.559789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34553 (local address 127.0.0.1:38518)
0927 02:51:56.559939 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.559975 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.560406 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.560630 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.560631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.560897 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.561050 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.561054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.562006 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.562007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.563042 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.563044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.563526 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.563527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.563588 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.564079 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.564086 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.565363 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.566909 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.566915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.566917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.567091 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.567292 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.567293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.567293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.567371 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.567680 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.567685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.567752 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.567830 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.568029 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":70,"thread_start_us":61,"threads_started":1}
I20260927 02:51:56.568195 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.559834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.0.98.62:34553)
0927 02:51:56.559960 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.559968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.560051 (+ 83us) server_negotiation.cc:407] Connection header received
0927 02:51:56.560520 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.560522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.560537 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.560617 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.561118 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.561923 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.563121 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.563430 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.563531 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.565434 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.565440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.565441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.565447 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.566846 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.567157 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.567160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.567160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.567239 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.567421 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.567421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.567422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.567624 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.567692 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.567857 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.567962 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260927 02:51:56.568809 22688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 02:51:56.569118 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.569014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34246)
0927 02:51:56.569052 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.569094 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 02:51:56.569240 22637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
I20260927 02:51:56.569569 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.569485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34260)
0927 02:51:56.569525 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.569548 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:51:56.569649 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45029: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:51:56.583455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45029
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34553,127.0.98.61:45029,127.0.98.60:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:51:56.618851 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } attempt: 1
I20260927 02:51:56.619284 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.619154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34262)
0927 02:51:56.619203 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.619255 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 02:51:56.619568 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.619488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34274)
0927 02:51:56.619523 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.619545 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:51:56.619652 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45029: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
W20260927 02:51:56.661793 22695 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:51:56.662006 22695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:56.662029 22695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:56.662055 22695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:56.663234 22695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:56.663275 22695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:56.663292 22695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:56.663307 22695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:56.663324 22695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:56.664783 22695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34553,127.0.98.61:45029,127.0.98.60:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:56.665015 22695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:56.665217 22695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:56.669620 22701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:56.669626 22700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:56.669626 22703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:56.669929 22695 server_base.cc:1034] running on GCE node
I20260927 02:51:56.670120 22695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:56.670349 22695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:56.671485 22695 hybrid_clock.cc:648] HybridClock initialized: now 1790477516671474 us; error 24 us; skew 500 ppm
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:56.673825 22695 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:51:56.674222 22695 webserver.cc:460] Webserver started at http://127.0.98.61:32803/ using document root <none> and password file <none>
I20260927 02:51:56.674392 22695 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:56.674428 22695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:56.674516 22695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:56.675518 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data/instance:
uuid: "741ed6ac1ba546118e4d3be1279dbd2d"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "14d1bca207ef8f142a99ea87262ae75a"
server_key_iv: "e41d42976743b554e3bf8b970db0ed1b"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.675806 22695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/wal/instance:
uuid: "741ed6ac1ba546118e4d3be1279dbd2d"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "14d1bca207ef8f142a99ea87262ae75a"
server_key_iv: "e41d42976743b554e3bf8b970db0ed1b"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.676932 22695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:56.677546 22710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:56.677784 22695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:56.677870 22695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/wal
uuid: "741ed6ac1ba546118e4d3be1279dbd2d"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "14d1bca207ef8f142a99ea87262ae75a"
server_key_iv: "e41d42976743b554e3bf8b970db0ed1b"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.677925 22695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:56.687779 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } attempt: 2
I20260927 02:51:56.688235 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.688104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34286)
0927 02:51:56.688156 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.688194 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:51:56.688576 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.688478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34298)
0927 02:51:56.688519 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.688547 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 02:51:56.688684 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45029: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45029: connect: Connection refused (error 111)
I20260927 02:51:56.691124 22695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:56.692790 22695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:56.692914 22695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:56.697243 22695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45029
I20260927 02:51:56.697428 22762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45029 every 8 connection(s)
I20260927 02:51:56.697713 22695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:51:56.698694 22763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:56.700752 22763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:56.707748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22695
I20260927 02:51:56.707854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-1/wal/instance
I20260927 02:51:56.708067 392 external_mini_cluster.cc:1428] Setting key 3efb96882dc5a53e00b3c0ad0c00cd70
I20260927 02:51:56.711071 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.701518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38532 (local address 127.0.98.62:34553)
0927 02:51:56.701574 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.701576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.701824 (+ 248us) server_negotiation.cc:407] Connection header received
0927 02:51:56.702402 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.702403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.702419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.702468 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.703286 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.703899 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.705205 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.705403 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.705460 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.707756 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.707758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.707758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.707765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.709424 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.709887 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.709888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.709889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.709952 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.710251 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.710252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.710252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.710393 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.710451 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.710891 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.710983 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 02:51:56.711555 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.701512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34553 (local address 127.0.0.1:38532)
0927 02:51:56.701679 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.701727 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.702295 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.702550 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.702554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.702904 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.703188 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.703195 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.704008 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.704009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.705005 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.705008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.705493 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.705494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.705606 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.706061 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.706070 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.707634 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.709518 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.709525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.709527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.709816 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.710028 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.710029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.710029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.710161 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.710478 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.710485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.710713 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.711163 (+ 450us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.711338 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260927 02:51:56.712116 22763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:51:56.722195 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.712388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34312)
0927 02:51:56.712438 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.712450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.712496 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.712726 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.712727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.712866 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.712922 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.712924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.714159 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.714160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.714965 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.714968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.715625 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.715626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.715702 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.716147 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.716153 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.719288 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.721246 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.721247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.721248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.721420 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.721583 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.721584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.721584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.721624 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.721931 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.721932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.721965 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.722013 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.722097 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 02:51:56.722471 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.712423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.0.98.61:45029)
0927 02:51:56.712572 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.712577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.712597 (+ 20us) server_negotiation.cc:407] Connection header received
0927 02:51:56.712620 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.712622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.712635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.712708 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.712999 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.714057 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.715049 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.715534 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.715600 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.719385 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.719394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.719395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.719402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.721158 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.721486 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.721489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.721489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.721533 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.721672 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.721672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.721673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.721839 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.721912 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.722323 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.722385 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260927 02:51:56.723086 22763 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:56.723466 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.723334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53378)
0927 02:51:56.723374 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.723442 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 02:51:56.723618 22714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
I20260927 02:51:56.724035 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.723955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53386)
0927 02:51:56.723990 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.724010 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:51:56.724180 22763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36827: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:51:56.733233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36827
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34553,127.0.98.61:45029,127.0.98.60:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:51:56.773444 22763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } attempt: 1
I20260927 02:51:56.774024 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.773855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53392)
0927 02:51:56.773925 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.773982 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 02:51:56.774463 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.774374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53398)
0927 02:51:56.774412 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.774442 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:51:56.774560 22763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36827: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
I20260927 02:51:56.779819 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } attempt: 3
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:51:56.787052 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.780231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.0.98.61:45029)
0927 02:51:56.780271 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.780273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.780286 (+ 13us) server_negotiation.cc:407] Connection header received
0927 02:51:56.780347 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.780348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.780358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.780431 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.780769 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.781571 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.782276 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.782604 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.782671 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.784559 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.784560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.784560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.784566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.785857 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.786251 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.786252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.786252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.786314 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.786518 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.786519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.786520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.786652 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.786714 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.786905 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.786977 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:51:56.787091 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.780186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34318)
0927 02:51:56.780245 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.780255 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.780300 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.780436 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.780436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.780586 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.780704 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.780706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.781668 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.781670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.782202 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.782204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.782684 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.782685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.782759 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.783213 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.783218 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.784476 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.785933 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.785934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.785935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.786160 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.786376 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.786377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.786377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.786440 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.786711 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.786712 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.786763 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.786877 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.786978 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 02:51:56.787525 22688 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:56.787868 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.787763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53404)
0927 02:51:56.787811 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.787843 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:51:56.788245 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.788157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53416)
0927 02:51:56.788200 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.788225 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 02:51:56.788336 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36827: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
W20260927 02:51:56.814981 22770 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:51:56.815192 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:56.815208 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:56.815228 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:56.816370 22770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:56.816404 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:56.816416 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:56.816427 22770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:56.816437 22770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:56.817867 22770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34553,127.0.98.61:45029,127.0.98.60:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:56.818064 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:56.818240 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 02:51:56.819458 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } attempt: 1
I20260927 02:51:56.819844 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.819730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53428)
0927 02:51:56.819780 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.819816 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 02:51:56.820209 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.820130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53442)
0927 02:51:56.820171 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.820194 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:51:56.820294 22688 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36827: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
W20260927 02:51:56.823040 22775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:56.823047 22776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:56.823055 22778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:56.823287 22770 server_base.cc:1034] running on GCE node
I20260927 02:51:56.823551 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:56.823812 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:56.824949 22770 hybrid_clock.cc:648] HybridClock initialized: now 1790477516824937 us; error 23 us; skew 500 ppm
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:56.827327 22770 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:51:56.827752 22770 webserver.cc:460] Webserver started at http://127.0.98.60:36291/ using document root <none> and password file <none>
I20260927 02:51:56.827950 22770 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:56.827996 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:56.828117 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:56.829103 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data/instance:
uuid: "c1adc338a6f24a56b688dee64f487883"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "5becbbe59bd0d4cc7aeeaa542c3dcfc6"
server_key_iv: "34808716cf7e80bc2ed8337c235bb56e"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.829427 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/wal/instance:
uuid: "c1adc338a6f24a56b688dee64f487883"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "5becbbe59bd0d4cc7aeeaa542c3dcfc6"
server_key_iv: "34808716cf7e80bc2ed8337c235bb56e"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.830592 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:56.831275 22785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:56.831408 22770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:56.831485 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/wal
uuid: "c1adc338a6f24a56b688dee64f487883"
format_stamp: "Formatted at 2026-09-27 02:51:56 on dist-test-slave-f4rg"
server_key: "5becbbe59bd0d4cc7aeeaa542c3dcfc6"
server_key_iv: "34808716cf7e80bc2ed8337c235bb56e"
server_key_version: "encryptionkey@0"
I20260927 02:51:56.831547 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:56.842708 22763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } attempt: 2
I20260927 02:51:56.843080 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.842976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53450)
0927 02:51:56.843025 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.843055 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:51:56.843358 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.843281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53456)
0927 02:51:56.843314 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.843337 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:51:56.843428 22763 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36827: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36827: connect: Connection refused (error 111)
I20260927 02:51:56.846333 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:56.848426 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:56.848551 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:56.852349 22770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36827
I20260927 02:51:56.852399 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36827 every 8 connection(s)
I20260927 02:51:56.852707 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:51:56.853623 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:56.855166 22838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:56.857915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 22770
I20260927 02:51:56.858022 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790477232139228-392-0/minicluster-data/master-2/wal/instance
I20260927 02:51:56.858209 392 external_mini_cluster.cc:1428] Setting key 71c691cfb1fafee650c4807e0617e5ec
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:51:56.863636 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.855824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.0.98.62:34553)
0927 02:51:56.855887 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.855889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.856044 (+ 155us) server_negotiation.cc:407] Connection header received
0927 02:51:56.856506 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.856508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.856519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.856569 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.857160 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.857651 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.858929 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.859123 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.859179 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.861075 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.861076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.861077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.861084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.862343 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.862697 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.862698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.862699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.862753 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.863027 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.863028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.863028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.863140 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.863205 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.863494 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.863552 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:51:56.863816 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.855776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34553 (local address 127.0.0.1:38544)
0927 02:51:56.855935 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.855970 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.856415 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.856634 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.856637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.856885 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.857075 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.857079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.857735 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.857736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.858848 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.858851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.859207 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.859208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.859328 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.859789 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.859798 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.860999 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.862412 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.862418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.862420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.862631 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.862823 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.862824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.862824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.862950 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.863214 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.863220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.863361 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.863442 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.863586 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260927 02:51:56.864482 22838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:56.865912 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.858572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.98.60:36827)
0927 02:51:56.858763 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.858768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.858791 (+ 23us) server_negotiation.cc:407] Connection header received
0927 02:51:56.858835 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.858838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.858864 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.858929 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.859486 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.860162 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.860887 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.861193 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.861249 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.863261 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.863268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.863269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.863275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.864785 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.865115 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.865118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.865118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.865175 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.865342 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.865342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.865343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.865457 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.865559 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.865674 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.865730 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:51:56.870599 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.864780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.98.61:45029)
0927 02:51:56.864823 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.864825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.864832 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:51:56.864858 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.864859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.864868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.864925 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.865113 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.865725 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.866397 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.866671 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.866738 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.868355 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.868356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.868356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.868363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.869701 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.869987 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.869988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.869988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.870039 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.870200 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.870200 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.870201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.870299 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.870346 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.870485 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.870538 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:51:56.870637 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.864710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45029 (local address 127.0.0.1:34320)
0927 02:51:56.864757 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.864768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.864810 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.864929 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.864929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.865022 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.865051 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.865052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.865798 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.865798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.866328 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.866330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.866736 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.866737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.866793 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.867169 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.867172 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.868286 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.869782 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.869783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.869783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.869928 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.870093 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.870093 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.870093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.870134 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.870353 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.870354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.870395 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.870468 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.870546 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:56.871140 22838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:56.873541 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.866586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.0.98.62:34553)
0927 02:51:56.866626 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.866628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.866646 (+ 18us) server_negotiation.cc:407] Connection header received
0927 02:51:56.866663 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.866664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.866672 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.866712 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.867266 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.867840 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.868595 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.868871 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.868919 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.871150 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.871151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.871151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.871157 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.872603 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.872934 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.872935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.872935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.872978 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.873126 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.873127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.873127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.873209 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.873349 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.873383 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.873451 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:51:56.877473 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.871362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53474)
0927 02:51:56.871396 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.871405 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.871443 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.871545 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.871546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.871685 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.871711 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.871712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.872526 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.872526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.873213 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.873215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.873629 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.873630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.873695 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.874037 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.874041 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.875294 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.876623 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.876624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.876624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.876810 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.876985 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.876986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.876987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.877035 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.877275 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.877276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.877306 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.877349 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.877414 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 02:51:56.877579 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.871391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.0.98.60:36827)
0927 02:51:56.871417 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.871419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.871428 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:56.871487 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.871487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.871496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.871543 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.871774 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.872445 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.873279 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.873557 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.873620 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.875361 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.875362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.875362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.875368 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.876555 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.876878 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.876879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.876880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.876929 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.877085 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.877086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.877086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.877220 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.877286 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.877376 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.877469 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":12}
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:51:56.878907 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: Bootstrap starting.
I20260927 02:51:56.879812 22838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:56.880059 22838 log.cc:824] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:56.880745 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: No bootstrap required, opened a new log
I20260927 02:51:56.881027 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.874046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.0.98.61:45029)
0927 02:51:56.874082 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.874084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.874134 (+ 50us) server_negotiation.cc:407] Connection header received
0927 02:51:56.874218 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.874219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.874227 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.874264 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.874736 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.875421 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.876303 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.876479 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.876527 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.878714 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.878715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.878716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.878722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.880073 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.880376 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.880377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.880377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.880429 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.880598 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.880598 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.880599 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.880704 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.880846 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.880880 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.880948 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:51:56.882311 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:56.882616 22838 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
I20260927 02:51:56.882737 22838 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1adc338a6f24a56b688dee64f487883, State: Initialized, Role: FOLLOWER
I20260927 02:51:56.882825 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
I20260927 02:51:56.883126 22838 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:56.883111 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.883209 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:56.883409 22662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "01a323ff4756466ba748bb7eb9fd8914"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38554
W20260927 02:51:56.884917 22855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:51:56.884972 22855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:56.885165 22770 master_runner.cc:186] Error getting master registration for 127.0.98.62:34553: OK, instance_id { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" instance_seqno: 1790477516555452 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:51:56.887878 22770 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:51:56.895429 22688 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } attempt: 2
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:51:56.902076 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.895743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53480)
0927 02:51:56.895798 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.895808 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.895847 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.895988 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.895989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.896152 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.896194 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.896195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.896784 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.896785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.897280 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.897281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.897584 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.897584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.897632 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.898097 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.898102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.899363 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.901012 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.901013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.901013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.901245 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.901424 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.901424 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.901425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.901485 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.901766 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.901767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.901815 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.901882 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.901975 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 02:51:56.902076 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.895839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.0.98.60:36827)
0927 02:51:56.895895 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.895897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.895903 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:51:56.895924 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.895925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.895933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.895977 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.896259 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.896718 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.897346 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.897528 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.897597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.899437 (+ 1840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.899438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.899438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.899445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.900927 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.901314 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.901315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.901315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.901369 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.901536 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.901536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.901536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.901710 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.901770 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.901909 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.901983 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:56.903798 22688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914: Bootstrap starting.
I20260927 02:51:56.904508 22688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:56.904816 22688 log.cc:824] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:56.905629 22688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914: No bootstrap required, opened a new log
I20260927 02:51:56.907023 22688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
I20260927 02:51:56.907162 22688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a323ff4756466ba748bb7eb9fd8914, State: Initialized, Role: FOLLOWER
I20260927 02:51:56.907240 22688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
I20260927 02:51:56.907490 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.907557 22688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:56.907586 22662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:51:56.907608 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:56.907748 22662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:56.908453 22662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
I20260927 02:51:56.909045 22812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01a323ff4756466ba748bb7eb9fd8914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c1adc338a6f24a56b688dee64f487883"
I20260927 02:51:56.909200 22812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:56.910058 22812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a323ff4756466ba748bb7eb9fd8914 in term 1.
I20260927 02:51:56.910298 22639 leader_election.cc:302] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a323ff4756466ba748bb7eb9fd8914, c1adc338a6f24a56b688dee64f487883; no voters:
I20260927 02:51:56.910421 22858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:56.910553 22858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [term 1 LEADER]: Becoming Leader. State: Replica: 01a323ff4756466ba748bb7eb9fd8914, State: Running, Role: LEADER
I20260927 02:51:56.910651 22858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
W20260927 02:51:56.911006 22869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 02:51:56.911072 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a323ff4756466ba748bb7eb9fd8914. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.911147 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:56.911158 22662 leader_election.cc:288] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [CANDIDATE]: Term 1 election: Requested vote from peers 741ed6ac1ba546118e4d3be1279dbd2d (127.0.98.61:45029), c1adc338a6f24a56b688dee64f487883 (127.0.98.60:36827)
W20260927 02:51:56.911429 22869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:56.912223 22873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:56.910269 22620 master_runner.cc:186] Error getting master registration for 127.0.98.61:45029: OK, instance_id { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" instance_seqno: 1790477516695826 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:51:56.912441 22620 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:51:56.912506 22873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:56.914392 22812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [term 1 FOLLOWER]: Refusing update from remote peer 01a323ff4756466ba748bb7eb9fd8914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:56.914759 22858 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:56.916041 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a323ff4756466ba748bb7eb9fd8914. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.916132 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:56.924290 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.924383 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:56.924798 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.924877 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:56.925154 22878 mvcc.cc:205] Tried to move back new op lower bound from 7333795909279375360 to 7333795909266894848. Current Snapshot: MvccSnapshot[applied={T|T < 7333795909279375360}]
I20260927 02:51:56.931772 22873 catalog_manager.cc:1347] Generated new cluster ID: ae01b1b0ca36408e96a4604a9af44c02
I20260927 02:51:56.931828 22873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 02:51:56.932577 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.913946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53490 (local address 127.0.98.60:36827)
0927 02:51:56.913983 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.913986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.915294 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 02:51:56.915453 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.915454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.915478 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.915509 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.917856 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.918600 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.925090 (+ 6490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.925393 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.925431 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.929329 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.929331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.929331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.929350 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.930934 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.931483 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.931484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.931485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.931543 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.931960 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.931961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.931961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.932102 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.932184 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.932421 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.932492 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:51:56.934505 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.914432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.0.98.62:34553)
0927 02:51:56.914466 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.914468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.915783 (+ 1315us) server_negotiation.cc:407] Connection header received
0927 02:51:56.916052 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.916053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.916075 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.916102 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.918731 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.919458 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.923861 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.924136 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.924189 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.929520 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.929522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.929522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.929799 (+ 277us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.931699 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.932767 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.932769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.932769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.932825 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.933636 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.933637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.933638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.933784 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.933870 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.934328 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.934403 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:51:56.934538 22763 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } attempt: 3
Sep 27 02:51:56 dist-test-slave-f4rg krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:51:56.939034 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.914869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.0.98.61:45029)
0927 02:51:56.914909 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.914911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.920238 (+ 5327us) server_negotiation.cc:407] Connection header received
0927 02:51:56.920566 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.920568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.920591 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.920621 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.924298 (+ 3677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.924969 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.927590 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.927907 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.927944 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.934963 (+ 7019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.934964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.934964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.934971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.936653 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.937978 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.937979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.937979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.938062 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.938338 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.938338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.938339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.938472 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.938556 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.938865 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.938941 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 02:51:56.941807 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.934780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36827 (local address 127.0.0.1:53492)
0927 02:51:56.934986 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:56.934996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:56.935041 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:56.935193 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:56.935194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:56.935337 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:56.935377 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.935379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.936190 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.936191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.936953 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:56.936954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.937336 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.937337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:56.937386 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.937738 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:56.937742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:56.939115 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:56.940749 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.940750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.940751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.940952 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.941167 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:56.941167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:56.941168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:56.941233 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:56.941512 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:56.941512 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:56.941564 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:51:56.941618 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 02:51:56.941705 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260927 02:51:56.942272 22842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.934950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.0.98.60:36827)
0927 02:51:56.935003 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.935004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.935028 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:51:56.935105 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.935106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.935116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.935159 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.935432 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.936107 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.937011 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.937261 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.937328 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.939181 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:56.939182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:56.939182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:56.939189 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:56.940643 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.941031 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.941032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.941032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.941096 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:56.941299 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:56.941300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:56.941300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:56.941428 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:56.941478 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.942106 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.942187 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 02:51:56.942957 22873 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:56.943643 22873 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:56.943697 22763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d: Bootstrap starting.
I20260927 02:51:56.944360 22763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:56.944608 22763 log.cc:824] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:56.945456 22763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d: No bootstrap required, opened a new log
I20260927 02:51:56.947280 22763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
I20260927 02:51:56.947405 22763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741ed6ac1ba546118e4d3be1279dbd2d, State: Initialized, Role: FOLLOWER
I20260927 02:51:56.947476 22763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } }
I20260927 02:51:56.947780 22763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:56.947881 22737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "01a323ff4756466ba748bb7eb9fd8914" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "741ed6ac1ba546118e4d3be1279dbd2d"
I20260927 02:51:56.947992 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:56.948793 22737 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a323ff4756466ba748bb7eb9fd8914 in term 1.
I20260927 02:51:56.949215 22733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [term 1 FOLLOWER]: Refusing update from remote peer 01a323ff4756466ba748bb7eb9fd8914: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:56.949532 22885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:56.951738 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.952138 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:56.952178 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.952216 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:56.953555 22903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a323ff4756466ba748bb7eb9fd8914. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.953634 22903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:56.954963 22873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914: Generated new TSK 0
I20260927 02:51:56.955118 22858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01a323ff4756466ba748bb7eb9fd8914" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01a323ff4756466ba748bb7eb9fd8914" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34553 } } peers { permanent_uuid: "741ed6ac1ba546118e4d3be1279dbd2d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45029 } } peers { permanent_uuid: "c1adc338a6f24a56b688dee64f487883" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36827 } } }
I20260927 02:51:56.955204 22858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a323ff4756466ba748bb7eb9fd8914 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:56.955405 22873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 02:51:56.956359 22899 catalog_manager.cc:1259] Loaded cluster ID: ae01b1b0ca36408e96a4604a9af44c02
I20260927 02:51:56.956403 22899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d: loading cluster ID for follower catalog manager: success
I20260927 02:51:56.957484 22899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d: acquiring CA information for follower catalog manager: success
I20260927 02:51:56.957787 22899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 741ed6ac1ba546118e4d3be1279dbd2d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 02:51:56.982717 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:56.981064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.0.98.60:36827)
0927 02:51:56.981106 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.981109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.982078 (+ 969us) server_negotiation.cc:407] Connection header received
0927 02:51:56.982177 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.982178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.982189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.982217 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.982638 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53498: BlockingRecv error: recv got EOF from 127.0.0.1:53498 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:51:56.983485 22691 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:56.981468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.0.98.62:34553)
0927 02:51:56.981526 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.981528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.982998 (+ 1470us) server_negotiation.cc:407] Connection header received
0927 02:51:56.983086 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.983087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.983097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.983122 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.983426 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38566: BlockingRecv error: recv got EOF from 127.0.0.1:38566 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 02:51:56.984278 22767 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:56.981833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.0.98.61:45029)
0927 02:51:56.981869 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.981870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.983711 (+ 1841us) server_negotiation.cc:407] Connection header received
0927 02:51:56.983832 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.983833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.983843 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.983870 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.984207 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34350: BlockingRecv error: recv got EOF from 127.0.0.1:34350 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:56.989621 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.986099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.0.98.61:45029)
0927 02:51:56.986145 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.986146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.986406 (+ 260us) server_negotiation.cc:407] Connection header received
0927 02:51:56.986516 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.986517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.986559 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.986602 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:56.986830 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.987614 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.988735 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.989003 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.989042 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.989257 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:56.989382 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:56.989429 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.989530 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.989548 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 02:51:56.993103 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:56.992321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.0.98.60:36827)
0927 02:51:56.992359 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.992360 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.992565 (+ 205us) server_negotiation.cc:407] Connection header received
0927 02:51:56.992675 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.992676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.992695 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.992723 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:56.993050 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53508: BlockingRecv error: recv got EOF from 127.0.0.1:53508 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:51:56.998433 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:56.995391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34366 (local address 127.0.98.61:45029)
0927 02:51:56.995426 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:56.995428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:56.995665 (+ 237us) server_negotiation.cc:407] Connection header received
0927 02:51:56.995757 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:56.995758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:56.995767 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:56.995793 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:56.995957 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.996620 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.997673 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:56.997946 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:56.997990 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:56.998186 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:56.998220 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:56.998246 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:56.998347 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 02:51:56.998365 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:51:57.001514 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.000741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53512 (local address 127.0.98.60:36827)
0927 02:51:57.000776 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.000778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.000998 (+ 220us) server_negotiation.cc:407] Connection header received
0927 02:51:57.001092 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.001093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.001103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.001131 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.001464 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53512: BlockingRecv error: recv got EOF from 127.0.0.1:53512 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:57.007992 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.004638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.0.98.61:45029)
0927 02:51:57.004688 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.004690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.004936 (+ 246us) server_negotiation.cc:407] Connection header received
0927 02:51:57.005032 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.005033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.005043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.005070 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.005237 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.005926 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.007062 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.007372 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.007418 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.007679 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.007712 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.007748 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.007898 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.007918 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 02:51:57.012056 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.010343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.0.98.60:36827)
0927 02:51:57.010383 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.010385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.010579 (+ 194us) server_negotiation.cc:407] Connection header received
0927 02:51:57.010690 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.010691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.010704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.010733 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.012002 (+ 1269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53520: BlockingRecv error: recv got EOF from 127.0.0.1:53520 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:57.020303 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.017157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.0.98.61:45029)
0927 02:51:57.017192 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.017194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.017412 (+ 218us) server_negotiation.cc:407] Connection header received
0927 02:51:57.017504 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.017505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.017515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.017541 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.017724 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.018437 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.019524 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.019824 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.019865 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.020056 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.020091 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.020117 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.020214 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.020231 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:51:57.023335 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.022500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53522 (local address 127.0.98.60:36827)
0927 02:51:57.022538 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.022540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.022776 (+ 236us) server_negotiation.cc:407] Connection header received
0927 02:51:57.022892 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.022893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.022905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.022934 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.023279 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53522: BlockingRecv error: recv got EOF from 127.0.0.1:53522 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:57.036072 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.032573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34392 (local address 127.0.98.61:45029)
0927 02:51:57.032621 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.032623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.032892 (+ 269us) server_negotiation.cc:407] Connection header received
0927 02:51:57.032990 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.032992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.033001 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.033027 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.033195 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.033941 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.035285 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.035570 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.035614 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.035821 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.035856 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.035880 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.035975 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.035996 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:51:57.039186 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.038359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.0.98.60:36827)
0927 02:51:57.038405 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.038407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.038617 (+ 210us) server_negotiation.cc:407] Connection header received
0927 02:51:57.038725 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.038726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.038738 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.038765 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.039132 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53532: BlockingRecv error: recv got EOF from 127.0.0.1:53532 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 02:51:57.060086 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.056496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.98.61:45029)
0927 02:51:57.056540 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.056542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.056910 (+ 368us) server_negotiation.cc:407] Connection header received
0927 02:51:57.057020 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.057021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.057031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.057058 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.057257 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.058002 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.059165 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.059533 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.059580 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.059791 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.059828 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.059853 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.059975 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.059997 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 02:51:57.063304 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.062480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.98.60:36827)
0927 02:51:57.062517 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.062519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.062749 (+ 230us) server_negotiation.cc:407] Connection header received
0927 02:51:57.062884 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.062885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.062897 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.062923 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.063247 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53548: BlockingRecv error: recv got EOF from 127.0.0.1:53548 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:57.098799 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.096533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.0.98.61:45029)
0927 02:51:57.096581 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.096583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.096735 (+ 152us) server_negotiation.cc:407] Connection header received
0927 02:51:57.096788 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.096789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.096800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.096847 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.096965 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.097459 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.098210 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.098393 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.098448 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.098575 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.098604 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.098642 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.098700 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.098719 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:51:57.103737 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.103109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.0.98.60:36827)
0927 02:51:57.103155 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.103157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.103265 (+ 108us) server_negotiation.cc:407] Connection header received
0927 02:51:57.103321 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.103322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.103334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.103379 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.103681 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53564: BlockingRecv error: recv got EOF from 127.0.0.1:53564 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:51:57.170976 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.168893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.0.98.61:45029)
0927 02:51:57.168948 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.168950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.169027 (+ 77us) server_negotiation.cc:407] Connection header received
0927 02:51:57.169076 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.169077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.169087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.169134 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.169249 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.169727 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.170418 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.170585 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.170626 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.170751 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.170775 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.170827 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.170887 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.170901 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 02:51:57.172979 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.172462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.0.98.60:36827)
0927 02:51:57.172530 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.172531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.172553 (+ 22us) server_negotiation.cc:407] Connection header received
0927 02:51:57.172606 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.172607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.172617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.172667 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.172924 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53572: BlockingRecv error: recv got EOF from 127.0.0.1:53572 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 02:51:57.304278 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.302106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.0.98.61:45029)
0927 02:51:57.302170 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.302173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.302242 (+ 69us) server_negotiation.cc:407] Connection header received
0927 02:51:57.302293 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.302294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.302303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.302365 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.302464 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.302985 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.303671 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.303854 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.303907 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.304052 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.304081 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.304121 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.304182 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.304200 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 02:51:57.306356 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.305839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.0.98.60:36827)
0927 02:51:57.305923 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.305925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.305934 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:57.305983 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.305984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.305993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.306038 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.306304 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53580: BlockingRecv error: recv got EOF from 127.0.0.1:53580 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 02:51:57.565649 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:57.563553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.0.98.61:45029)
0927 02:51:57.563606 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.563608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.563764 (+ 156us) server_negotiation.cc:407] Connection header received
0927 02:51:57.563797 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.563798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.563808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.563858 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:57.563966 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.564429 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.565130 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:57.565302 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:57.565345 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:57.565454 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:57.565480 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:57.565514 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:57.565561 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 02:51:57.565576 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 02:51:57.567811 22842 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:57.567346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.0.98.60:36827)
0927 02:51:57.567389 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:57.567390 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:57.567398 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:57.567420 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:57.567421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:57.567431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:57.567475 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:57.567758 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53592: BlockingRecv error: recv got EOF from 127.0.0.1:53592 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:51:57.885363 22855 catalog_manager.cc:1259] Loaded cluster ID: ae01b1b0ca36408e96a4604a9af44c02
I20260927 02:51:57.885432 22855 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: loading cluster ID for follower catalog manager: success
I20260927 02:51:57.886226 22855 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: acquiring CA information for follower catalog manager: success
I20260927 02:51:57.886523 22855 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1adc338a6f24a56b688dee64f487883: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:51:58.083124 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:58.080810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.0.98.61:45029)
0927 02:51:58.080931 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:58.080933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:58.080968 (+ 35us) server_negotiation.cc:407] Connection header received
0927 02:51:58.081043 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:58.081044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:58.081055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:58.081099 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:58.081194 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.081697 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.082461 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.082672 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.082715 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:58.082836 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:58.082861 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:58.082894 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:58.083005 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 02:51:58.083031 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260927 02:51:58.086655 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:58.084588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.98.60:36827)
0927 02:51:58.084708 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:58.084710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:58.084719 (+ 9us) server_negotiation.cc:407] Connection header received
0927 02:51:58.084803 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:58.084804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:58.084852 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:58.084918 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:58.084990 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.085456 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.086080 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.086270 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.086311 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:58.086397 (+ 86us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:58.086498 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:58.086547 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:58.086560 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 02:51:58.086574 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":46,"threads_started":1}
I20260927 02:51:58.091068 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:58.088057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.0.98.62:34553)
0927 02:51:58.088220 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:58.088222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:58.088339 (+ 117us) server_negotiation.cc:407] Connection header received
0927 02:51:58.088441 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:58.088443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:58.088495 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:58.088569 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:58.088708 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.089183 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.090186 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.090504 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.090562 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:58.090729 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:58.090842 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:58.090908 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:58.090950 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 02:51:58.090970 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":52,"threads_started":1}
I20260927 02:51:58.091224 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:58.088174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.0.98.61:45029)
0927 02:51:58.088264 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:58.088266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:58.088278 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:51:58.088367 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:58.088368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:58.088377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:58.088417 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:58.088554 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.089288 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.090214 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.090496 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.090562 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:58.090841 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:58.090869 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:58.090915 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:58.091126 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 02:51:58.091149 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 02:51:58.092147 23027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:58.088147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.98.60:36827)
0927 02:51:58.088184 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:58.088186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:58.088320 (+ 134us) server_negotiation.cc:407] Connection header received
0927 02:51:58.088383 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:58.088384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:58.088392 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:58.088433 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:51:58.088574 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.089294 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.090947 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:58.091748 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:58.091805 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:58.091937 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:51:58.091970 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:51:58.092012 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:58.092057 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 02:51:58.092077 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:51:58.092974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22620
I20260927 02:51:58.097570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22695
I20260927 02:51:58.101758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 22770
2026-09-27T02:51:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2706 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 02:51:58 dist-test-slave-f4rg krb5kdc[23038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 02:51:58 dist-test-slave-f4rg krb5kdc[23038](info): set up 2 sockets
Sep 27 02:51:58 dist-test-slave-f4rg krb5kdc[23038](info): commencing operation
krb5kdc: starting...
W20260927 02:51:59.038849 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/test-user.keytab.
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T02:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T02:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 02:51:59.145288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36337,127.0.98.61:40583,127.0.98.60:34343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 02:51:59.222683 23051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:51:59.222921 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:59.222937 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:59.222957 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:59.224119 23051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:59.224161 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:59.224175 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:59.224189 23051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:59.224211 23051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:59.225716 23051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36337,127.0.98.61:40583,127.0.98.60:34343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:59.225958 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:59.226177 23051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:59.231456 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:59.231482 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:59.231606 23051 server_base.cc:1034] running on GCE node
W20260927 02:51:59.231434 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:59.231961 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:59.232223 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:59.233355 23051 hybrid_clock.cc:648] HybridClock initialized: now 1790477519233340 us; error 32 us; skew 500 ppm
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:59.242787 23051 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 02:51:59.243323 23051 webserver.cc:460] Webserver started at http://127.0.98.62:41593/ using document root <none> and password file <none>
I20260927 02:51:59.243563 23051 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:59.243626 23051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:59.243762 23051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:59.245096 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data/instance:
uuid: "9dbb55830dc24cb6b106760407798888"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "03da9eed2eb395177ce113e62da066ae"
server_key_iv: "497395e6b014a7892af061ac93a1a0e0"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.245499 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/wal/instance:
uuid: "9dbb55830dc24cb6b106760407798888"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "03da9eed2eb395177ce113e62da066ae"
server_key_iv: "497395e6b014a7892af061ac93a1a0e0"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.246730 23051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 02:51:59.247478 23066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:59.247674 23051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:59.247762 23051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/wal
uuid: "9dbb55830dc24cb6b106760407798888"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "03da9eed2eb395177ce113e62da066ae"
server_key_iv: "497395e6b014a7892af061ac93a1a0e0"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.247833 23051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:59.257759 23051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:59.259613 23051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:59.259732 23051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:59.263490 23051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36337
I20260927 02:51:59.263536 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36337 every 8 connection(s)
I20260927 02:51:59.263850 23051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/data/info.pb
I20260927 02:51:59.264747 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:59.266184 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:59.269310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 23051
I20260927 02:51:59.269454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-0/wal/instance
I20260927 02:51:59.269644 392 external_mini_cluster.cc:1428] Setting key 29f0b4c70499bf3d56cb39cc078a4c84
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:59.275249 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.266763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36337 (local address 127.0.0.1:60620)
0927 02:51:59.266926 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.266965 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.267402 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.267641 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.267643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.267885 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.268036 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.268040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.268968 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.268969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.269807 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.269809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.270460 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.270461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.270520 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.271050 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.271058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.272337 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.273917 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.273923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.273925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.274096 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.274306 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.274308 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.274308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.274381 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.274675 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.274680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.274758 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.274865 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.275075 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":87,"thread_start_us":92,"threads_started":1}
I20260927 02:51:59.275254 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.266853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.0.98.62:36337)
0927 02:51:59.266992 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.266996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.267038 (+ 42us) server_negotiation.cc:407] Connection header received
0927 02:51:59.267527 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.267528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.267544 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.267626 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.268104 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.268898 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.269893 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.270351 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.270460 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.272414 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.272420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.272421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.272427 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.273835 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.274163 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.274166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.274166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.274252 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.274432 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.274432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.274433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.274621 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.274688 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.274892 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.275021 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260927 02:51:59.276017 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 02:51:59.276306 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.276207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59692)
0927 02:51:59.276244 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.276284 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 02:51:59.276396 23067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
I20260927 02:51:59.276772 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.276685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59704)
0927 02:51:59.276725 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.276752 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 02:51:59.276871 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40583: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 02:51:59.291867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40583
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36337,127.0.98.61:40583,127.0.98.60:34343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:51:59.326090 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } attempt: 1
I20260927 02:51:59.326543 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.326424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59716)
0927 02:51:59.326479 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.326513 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 02:51:59.326892 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.326796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59732)
0927 02:51:59.326846 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.326871 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 02:51:59.326980 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40583: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
W20260927 02:51:59.369585 23126 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:51:59.369794 23126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:59.369817 23126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:59.369840 23126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:59.370956 23126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:59.370993 23126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:59.371011 23126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:59.371027 23126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:59.371042 23126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:59.372488 23126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36337,127.0.98.61:40583,127.0.98.60:34343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:59.372728 23126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:59.372929 23126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:59.377332 23131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:59.377338 23132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:59.377476 23134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:59.377609 23126 server_base.cc:1034] running on GCE node
I20260927 02:51:59.377873 23126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:59.378144 23126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:59.379284 23126 hybrid_clock.cc:648] HybridClock initialized: now 1790477519379271 us; error 27 us; skew 500 ppm
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:59.381651 23126 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 02:51:59.382084 23126 webserver.cc:460] Webserver started at http://127.0.98.61:45011/ using document root <none> and password file <none>
I20260927 02:51:59.382272 23126 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:59.382316 23126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:59.382438 23126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:59.383509 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data/instance:
uuid: "13b8888f57ea4282b18789ede8285eaa"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "04fe698f35112a84687b040c35ee66ae"
server_key_iv: "a7866cec329d48c075587180cfe1188a"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.383872 23126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/wal/instance:
uuid: "13b8888f57ea4282b18789ede8285eaa"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "04fe698f35112a84687b040c35ee66ae"
server_key_iv: "a7866cec329d48c075587180cfe1188a"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.385082 23126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:59.385775 23141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:59.385900 23126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:59.385978 23126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/wal
uuid: "13b8888f57ea4282b18789ede8285eaa"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "04fe698f35112a84687b040c35ee66ae"
server_key_iv: "a7866cec329d48c075587180cfe1188a"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.386039 23126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:59.395105 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } attempt: 2
I20260927 02:51:59.395478 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.395339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59740)
0927 02:51:59.395399 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.395441 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 02:51:59.395627 23126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:59.395818 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.395708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59748)
0927 02:51:59.395764 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.395793 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 02:51:59.395898 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40583: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40583: connect: Connection refused (error 111)
I20260927 02:51:59.397349 23126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:59.397464 23126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:59.401023 23126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40583
I20260927 02:51:59.401051 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40583 every 8 connection(s)
I20260927 02:51:59.401353 23126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/data/info.pb
I20260927 02:51:59.402215 23194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:59.403661 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:59.405949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 23126
I20260927 02:51:59.406059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-1/wal/instance
I20260927 02:51:59.406258 392 external_mini_cluster.cc:1428] Setting key 2ed443a51f3b00ae42512e261fc44c84
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:59.411938 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.404237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.0.98.62:36337)
0927 02:51:59.404301 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.404303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.404476 (+ 173us) server_negotiation.cc:407] Connection header received
0927 02:51:59.404903 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.404904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.404913 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.404959 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.405474 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.405933 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.407280 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.407472 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.407539 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.409461 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.409462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.409463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.409469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.410753 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.411132 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.411134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.411134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.411184 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.411384 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.411385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.411385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.411483 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.411535 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.411778 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.411854 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 02:51:59.412099 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.404217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36337 (local address 127.0.0.1:60628)
0927 02:51:59.404373 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.404407 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.404833 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.405011 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.405014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.405236 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.405414 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.405418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.406008 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.406009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.407211 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.407213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.407577 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.407578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.407689 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.408100 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.408108 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.409384 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.410849 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.410856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.410858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.411069 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.411235 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.411235 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.411236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.411334 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.411534 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.411539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.411667 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.411752 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.411889 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260927 02:51:59.412544 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:51:59.419173 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.412741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59760)
0927 02:51:59.412776 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.412786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.412820 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.413091 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.413092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.413195 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.413236 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.413238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.414010 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.414011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.414492 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.414494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.414978 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.414979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.415030 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.415394 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.415399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.416528 (+ 1129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.418284 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.418285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.418285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.418429 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.418613 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.418613 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.418613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.418654 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.418937 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.418938 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.418970 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.419040 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.419113 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 02:51:59.419274 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.412799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.0.98.61:40583)
0927 02:51:59.412948 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.412952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.412973 (+ 21us) server_negotiation.cc:407] Connection header received
0927 02:51:59.412994 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.412996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.413008 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.413089 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.413292 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.413954 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.414562 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.414924 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.414977 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.416622 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.416630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.416630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.416637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.418207 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.418497 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.418500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.418501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.418558 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.418688 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.418689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.418689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.418877 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.418963 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.419072 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.419178 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260927 02:51:59.419909 23194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:59.420207 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.420113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37144)
0927 02:51:59.420148 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.420188 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 02:51:59.420336 23142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
I20260927 02:51:59.420702 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.420620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37146)
0927 02:51:59.420657 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.420687 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 02:51:59.420796 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 02:51:59.429328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
/tmp/dist-test-taskc9ab2Y/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34343
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36337,127.0.98.61:40583,127.0.98.60:34343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 02:51:59.469992 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } attempt: 1
I20260927 02:51:59.470432 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.470321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37158)
0927 02:51:59.470374 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.470405 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 02:51:59.470793 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.470712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37174)
0927 02:51:59.470755 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.470778 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:51:59.470914 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
I20260927 02:51:59.487052 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } attempt: 3
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:51:59.493322 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.487430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.0.98.61:40583)
0927 02:51:59.487487 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.487489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.487499 (+ 10us) server_negotiation.cc:407] Connection header received
0927 02:51:59.487535 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.487536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.487548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.487587 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.487896 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.488391 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.489069 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.489277 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.489332 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.491089 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.491090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.491091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.491097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.492320 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.492694 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.492695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.492695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.492758 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.492931 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.492931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.492931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.493021 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.493060 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.493206 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.493262 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 02:51:59.493379 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.487369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59776)
0927 02:51:59.487417 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.487428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.487472 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.487606 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.487607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.487757 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.487823 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.487825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.488467 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.488468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.489003 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.489005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.489336 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.489337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.489403 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.489808 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.489813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.490994 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.492396 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.492397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.492398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.492619 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.492810 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.492811 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.492811 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.492878 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.493068 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.493069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.493121 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.493193 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.493287 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:51:59.493793 23119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:59.494096 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.493991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37176)
0927 02:51:59.494027 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.494067 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 02:51:59.494496 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.494416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37182)
0927 02:51:59.494449 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.494475 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:51:59.494577 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
W20260927 02:51:59.508168 23201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 02:51:59.508368 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 02:51:59.508383 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 02:51:59.508400 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 02:51:59.509439 23201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 02:51:59.509472 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 02:51:59.509485 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 02:51:59.509496 23201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 02:51:59.509507 23201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 02:51:59.510931 23201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36337,127.0.98.61:40583,127.0.98.60:34343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 01:31:55 UTC on ccycloud.quasar-nivtoa.root.comops.site
build id 4675562
I20260927 02:51:59.511118 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 02:51:59.511279 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 02:51:59.515579 23209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:59.515590 23206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 02:51:59.515592 23207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 02:51:59.515838 23201 server_base.cc:1034] running on GCE node
I20260927 02:51:59.516054 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 02:51:59.516328 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 02:51:59.517473 23201 hybrid_clock.cc:648] HybridClock initialized: now 1790477519517464 us; error 30 us; skew 500 ppm
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 02:51:59.519791 23201 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 02:51:59.520156 23201 webserver.cc:460] Webserver started at http://127.0.98.60:45129/ using document root <none> and password file <none>
I20260927 02:51:59.520334 23201 fs_manager.cc:359] Metadata directory not provided
I20260927 02:51:59.520370 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 02:51:59.520457 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 02:51:59.521399 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data/instance:
uuid: "b29a861275a44622971f6aa080ee6679"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "aaf3aa1aba83dc0fe71d827f2e184b78"
server_key_iv: "0df1044e17793af97f09fa2bad1e747c"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.521667 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/wal/instance:
uuid: "b29a861275a44622971f6aa080ee6679"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "aaf3aa1aba83dc0fe71d827f2e184b78"
server_key_iv: "0df1044e17793af97f09fa2bad1e747c"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.522771 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 02:51:59.523468 23216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 02:51:59.523626 23201 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 02:51:59.523710 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/wal
uuid: "b29a861275a44622971f6aa080ee6679"
format_stamp: "Formatted at 2026-09-27 02:51:59 on dist-test-slave-f4rg"
server_key: "aaf3aa1aba83dc0fe71d827f2e184b78"
server_key_iv: "0df1044e17793af97f09fa2bad1e747c"
server_key_version: "encryptionkey@0"
I20260927 02:51:59.523764 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 02:51:59.525722 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } attempt: 1
I20260927 02:51:59.526086 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.525965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37186)
0927 02:51:59.526017 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.526054 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 02:51:59.526432 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.526343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37190)
0927 02:51:59.526385 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.526412 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 02:51:59.526520 23119 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
I20260927 02:51:59.539047 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } attempt: 2
I20260927 02:51:59.539453 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.539338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37204)
0927 02:51:59.539392 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.539425 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 02:51:59.539729 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.539654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37208)
0927 02:51:59.539688 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.539711 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 02:51:59.539799 23194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34343: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34343: connect: Connection refused (error 111)
I20260927 02:51:59.544406 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 02:51:59.546058 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 02:51:59.546180 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 02:51:59.550017 23201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34343
I20260927 02:51:59.550045 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34343 every 8 connection(s)
I20260927 02:51:59.550365 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/data/info.pb
I20260927 02:51:59.551329 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 02:51:59.552793 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:59.553511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu as pid 23201
I20260927 02:51:59.553611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskc9ab2Y/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790477232139228-392-0/minicluster-data/master-2/wal/instance
I20260927 02:51:59.553784 392 external_mini_cluster.cc:1428] Setting key 80d9803090a9f625cd37a85504326152
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:51:59.561913 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.553426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.0.98.62:36337)
0927 02:51:59.553496 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.553498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.553670 (+ 172us) server_negotiation.cc:407] Connection header received
0927 02:51:59.554225 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.554228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.554248 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.554302 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.554924 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.555454 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.556545 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.556834 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.556899 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.559157 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.559159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.559160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.559167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.560563 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.561011 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.561013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.561013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.561068 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.561293 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.561294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.561294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.561406 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.561468 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.561751 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.561824 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 02:51:59.562095 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.553367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36337 (local address 127.0.0.1:60640)
0927 02:51:59.553532 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.553572 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.554092 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.554382 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.554386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.554667 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.554847 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.554854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.555521 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.555522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.556477 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.556479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.556937 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.556938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.557064 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.557556 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.557563 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.559077 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.560690 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.560696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.560698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.560949 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.561136 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.561137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.561137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.561236 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.561463 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.561467 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.561604 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.561713 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.561865 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":104,"thread_start_us":81,"threads_started":1}
I20260927 02:51:59.562095 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.554110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.0.98.60:34343)
0927 02:51:59.554317 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.554321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.554335 (+ 14us) server_negotiation.cc:407] Connection header received
0927 02:51:59.554375 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.554376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.554394 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.554468 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.555081 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.556099 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.556925 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.557329 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.557381 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.559564 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.559572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.559572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.559578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.561016 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.561328 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.561331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.561331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.561375 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.561533 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.561534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.561534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.561671 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.561805 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.561877 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.561935 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":87,"threads_started":1}
I20260927 02:51:59.562687 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 02:51:59.568437 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.562912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40583 (local address 127.0.0.1:59782)
0927 02:51:59.562939 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.562949 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.562982 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.563124 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.563124 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.563245 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.563272 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.563273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.563832 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.563833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.564285 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.564286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.564607 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.564608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.564660 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.564987 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.564992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.566065 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.567585 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.567586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.567586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.567727 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.567928 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.567929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.567929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.567968 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.568172 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.568173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.568213 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.568276 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.568348 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260927 02:51:59.568450 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.562994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.0.98.61:40583)
0927 02:51:59.563034 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.563035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.563043 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:59.563063 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.563064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.563072 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.563115 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.563332 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.563773 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.564346 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.564555 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.564605 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.566149 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.566150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.566150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.566156 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.567505 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.567811 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.567812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.567813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.567869 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.568004 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.568004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.568005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.568119 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.568163 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.568304 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.568376 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:51:59.568742 23269 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } has no permanent_uuid. Determining permanent_uuid...
I20260927 02:51:59.569114 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.562857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.0.98.62:36337)
0927 02:51:59.562889 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.562890 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.562916 (+ 26us) server_negotiation.cc:407] Connection header received
0927 02:51:59.563025 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.563026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.563035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.563077 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.563474 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.563929 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.564526 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.564715 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.564764 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.566724 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.566725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.566725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.566732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.567967 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.568254 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.568255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.568255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.568307 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.568471 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.568471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.568642 (+ 171us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.568763 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.568905 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.568967 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.569033 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 02:51:59.575002 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.568947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37234)
0927 02:51:59.568986 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.568994 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.569022 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.569160 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.569161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.569295 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.569324 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.569326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.570011 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.570012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.570590 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.570592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.571028 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.571029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.571081 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.571380 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.571384 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.572630 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.574051 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.574052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.574052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.574205 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.574391 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.574391 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.574392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.574461 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.574687 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.574688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.574730 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.574797 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.574901 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:51:59.575011 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.568999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.98.60:34343)
0927 02:51:59.569031 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.569033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.569039 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:51:59.569091 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.569092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.569101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.569418 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.569448 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.569947 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.570654 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.570973 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.571016 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.572704 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.572705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.572705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.572711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.573982 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.574274 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.574275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.574275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.574332 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.574518 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.574519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.574519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.574629 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.574688 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.574844 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.574944 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":17}
I20260927 02:51:59.576272 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.569657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.98.61:40583)
0927 02:51:59.569689 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.569691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.569731 (+ 40us) server_negotiation.cc:407] Connection header received
0927 02:51:59.569799 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.569800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.569807 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.569848 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.570328 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.570824 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.571581 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.571771 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.571835 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.573904 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.573905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.573905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.573911 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.575381 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.575668 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.575669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.575669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.575721 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.575855 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.575855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.575856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.575959 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.576092 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.576126 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.576192 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 02:51:59.576609 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: Bootstrap starting.
I20260927 02:51:59.577471 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 02:51:59.577533 23269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:59.577807 23269 log.cc:824] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:59.578559 23269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: No bootstrap required, opened a new log
I20260927 02:51:59.578550 23093 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9dbb55830dc24cb6b106760407798888"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60648
I20260927 02:51:59.580585 23269 raft_consensus.cc:348] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.580734 23269 raft_consensus.cc:729] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29a861275a44622971f6aa080ee6679, State: Initialized, Role: FOLLOWER
I20260927 02:51:59.580816 23269 consensus_queue.cc:260] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.581064 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.581123 23269 sys_catalog.cc:564] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:59.581167 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:51:59.582640 23286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:51:59.582695 23286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:59.582975 23201 master_runner.cc:186] Error getting master registration for 127.0.98.62:36337: OK, instance_id { permanent_uuid: "9dbb55830dc24cb6b106760407798888" instance_seqno: 1790477519262389 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:51:59.585641 23201 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:51:59.601646 23119 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } attempt: 2
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:51:59.608341 23121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.601942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37236)
0927 02:51:59.602003 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.602013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.602058 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.602183 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.602184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.602369 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.602418 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.602420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.603068 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.603069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.603558 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.603560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.603867 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.603868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.603919 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.604320 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.604325 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.605676 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.607298 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.607299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.607300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.607490 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.607685 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.607686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.607686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.607745 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.608049 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.608049 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.608094 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.608160 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.608243 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 02:51:59.608353 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.602009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.0.98.60:34343)
0927 02:51:59.602063 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.602065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.602089 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:51:59.602111 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.602112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.602129 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.602170 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.602482 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.603002 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.603631 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.603812 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.603863 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.605783 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.605784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.605785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.605791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.607207 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.607570 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.607571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.607571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.607629 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.607799 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.607800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.607800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.607989 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.608048 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.608189 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.608263 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 02:51:59.609582 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888: Bootstrap starting.
I20260927 02:51:59.610162 23119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:59.610407 23119 log.cc:824] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:59.611063 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888: No bootstrap required, opened a new log
I20260927 02:51:59.612004 23119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.612109 23119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dbb55830dc24cb6b106760407798888, State: Initialized, Role: FOLLOWER
I20260927 02:51:59.612172 23119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.612397 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.612450 23119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:59.612473 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:51:59.613955 23300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 02:51:59.614001 23300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:59.613576 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 02:51:59.614192 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:59.615080 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.615324 23093 leader_election.cc:288] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [CANDIDATE]: Term 1 election: Requested vote from peers 13b8888f57ea4282b18789ede8285eaa (127.0.98.61:40583), b29a861275a44622971f6aa080ee6679 (127.0.98.60:34343)
I20260927 02:51:59.615929 23243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbb55830dc24cb6b106760407798888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b29a861275a44622971f6aa080ee6679"
I20260927 02:51:59.616046 23243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:59.616899 23243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbb55830dc24cb6b106760407798888 in term 1.
I20260927 02:51:59.617218 23067 leader_election.cc:302] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dbb55830dc24cb6b106760407798888, b29a861275a44622971f6aa080ee6679; no voters:
I20260927 02:51:59.617383 23289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 02:51:59.617532 23289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [term 1 LEADER]: Becoming Leader. State: Replica: 9dbb55830dc24cb6b106760407798888, State: Running, Role: LEADER
I20260927 02:51:59.617662 23289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.618439 23051 master_runner.cc:186] Error getting master registration for 127.0.98.61:40583: OK, instance_id { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" instance_seqno: 1790477519400036 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 02:51:59.620627 23051 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 02:51:59.622836 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbb55830dc24cb6b106760407798888. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.622936 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:59.626382 23308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 02:51:59.626628 23308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 02:51:59.628335 23243 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [term 1 FOLLOWER]: Refusing update from remote peer 9dbb55830dc24cb6b106760407798888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:59.628804 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:59.630781 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbb55830dc24cb6b106760407798888. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.630888 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:59.630908 23194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } attempt: 3
I20260927 02:51:59.637034 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.637145 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:59.637225 23312 mvcc.cc:205] Tried to move back new op lower bound from 7333795920395120640 to 7333795920354693120. Current Snapshot: MvccSnapshot[applied={T|T < 7333795920395120640}]
I20260927 02:51:59.637777 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.637856 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:59.637908 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.623084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.0.98.61:40583)
0927 02:51:59.623133 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.623135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.623314 (+ 179us) server_negotiation.cc:407] Connection header received
0927 02:51:59.623494 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.623495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.623506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.623550 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.624802 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.625545 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.627135 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.627453 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.627495 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.632937 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.632939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.632939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.632946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.634694 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.636023 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.636024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.636024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.636078 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.636513 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.636513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.636514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.636650 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.637151 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.637754 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.637824 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:51:59.638660 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.623289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.98.60:34343)
0927 02:51:59.623324 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.623325 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.626224 (+ 2899us) server_negotiation.cc:407] Connection header received
0927 02:51:59.626310 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.626312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.626322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.626380 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.627107 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.627828 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.628813 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.629062 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.629098 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.633207 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.633209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.633209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.633216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.635482 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.636271 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.636272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.636272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.636320 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.637348 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.637349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.637350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.637472 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.637531 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.638495 (+ 964us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.638571 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 02:51:59.639916 23308 catalog_manager.cc:1347] Generated new cluster ID: 3ccd79debd764d83aa43291fdf0b0b2f
I20260927 02:51:59.639959 23308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 02:51:59 dist-test-slave-f4rg krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790477519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 02:51:59.642563 23313 mvcc.cc:205] Tried to move back new op lower bound from 7333795920395120640 to 7333795920354693120. Current Snapshot: MvccSnapshot[applied={T|T < 7333795920395120640 or (T in {7333795920395120640})}]
I20260927 02:51:59.644721 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.623603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.0.98.62:36337)
0927 02:51:59.623648 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.623649 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.628624 (+ 4975us) server_negotiation.cc:407] Connection header received
0927 02:51:59.628709 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.628710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.628719 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.628745 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.629412 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.630062 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.631392 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.631680 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.631715 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.634360 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.634361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.634361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.634368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.636089 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.643090 (+ 7001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.643092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.643092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.643168 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.643844 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.643845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.643846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.643981 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.644080 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.644568 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.644637 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 02:51:59.646786 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.631141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34343 (local address 127.0.0.1:37264)
0927 02:51:59.631183 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 02:51:59.631192 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 02:51:59.631223 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 02:51:59.635015 (+ 3792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 02:51:59.635016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 02:51:59.635175 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 02:51:59.635213 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.635215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.636879 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.636881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.637669 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 02:51:59.637672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.638151 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.638152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 02:51:59.638206 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.638701 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 02:51:59.638706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 02:51:59.640598 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 02:51:59.645398 (+ 4800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.645399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.645399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.645590 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.646106 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 02:51:59.646107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 02:51:59.646107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 02:51:59.646182 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 02:51:59.646518 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 02:51:59.646519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 02:51:59.646572 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 02:51:59.646611 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 02:51:59.646697 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 02:51:59.647534 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:51:59.631181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.0.98.60:34343)
0927 02:51:59.634880 (+ 3699us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.634882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.634890 (+ 8us) server_negotiation.cc:407] Connection header received
0927 02:51:59.634908 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.634908 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.634917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.635275 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.635303 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.636048 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.637759 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:51:59.638074 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:51:59.638335 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:51:59.640796 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 02:51:59.640797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 02:51:59.640798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 02:51:59.640803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 02:51:59.645205 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.645666 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.645667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.645667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.645994 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 02:51:59.646252 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 02:51:59.646253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 02:51:59.646253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 02:51:59.646441 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 02:51:59.647189 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0927 02:51:59.647251 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 02:51:59.647347 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3678,"thread_start_us":70,"threads_started":1}
I20260927 02:51:59.649036 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: Bootstrap starting.
I20260927 02:51:59.649683 23194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: Neither blocks nor log segments found. Creating new log.
I20260927 02:51:59.649924 23194 log.cc:824] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: Log is configured to *not* fsync() on all Append() calls
I20260927 02:51:59.650902 23194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: No bootstrap required, opened a new log
I20260927 02:51:59.652705 23194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.652818 23194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b8888f57ea4282b18789ede8285eaa, State: Initialized, Role: FOLLOWER
I20260927 02:51:59.652880 23194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } }
I20260927 02:51:59.653148 23194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 02:51:59.654564 23168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9dbb55830dc24cb6b106760407798888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "13b8888f57ea4282b18789ede8285eaa"
I20260927 02:51:59.654863 23167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 02:51:59.655721 23167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [term 1 FOLLOWER]: Refusing update from remote peer 9dbb55830dc24cb6b106760407798888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 02:51:59.655972 23289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 02:51:59.656857 23168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dbb55830dc24cb6b106760407798888 in term 1.
I20260927 02:51:59.658294 23332 mvcc.cc:205] Tried to move back new op lower bound from 7333795920395120640 to 7333795920354693120. Current Snapshot: MvccSnapshot[applied={T|T < 7333795920395120640}]
W20260927 02:51:59.658514 23330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 02:51:59.658484 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.658679 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:59.658764 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dbb55830dc24cb6b106760407798888. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.658885 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [sys.catalog]: This master's current role is: FOLLOWER
I20260927 02:51:59.658954 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.659044 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa [sys.catalog]: This master's current role is: FOLLOWER
W20260927 02:51:59.659124 23330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 02:51:59.668599 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9dbb55830dc24cb6b106760407798888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dbb55830dc24cb6b106760407798888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36337 } } peers { permanent_uuid: "13b8888f57ea4282b18789ede8285eaa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40583 } } peers { permanent_uuid: "b29a861275a44622971f6aa080ee6679" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34343 } } }
I20260927 02:51:59.668708 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888 [sys.catalog]: This master's current role is: LEADER
I20260927 02:51:59.670876 23308 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 02:51:59.671630 23308 catalog_manager.cc:1504] Loading token signing keys...
I20260927 02:51:59.686877 23308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dbb55830dc24cb6b106760407798888: Generated new TSK 0
I20260927 02:51:59.687227 23308 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 02:51:59.755746 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.750469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.0.98.62:36337)
0927 02:51:59.750514 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.750516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.755004 (+ 4488us) server_negotiation.cc:407] Connection header received
0927 02:51:59.755139 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.755141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.755153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.755184 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.755670 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60654: BlockingRecv error: recv got EOF from 127.0.0.1:60654 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 02:51:59.759905 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.759131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.0.98.61:40583)
0927 02:51:59.759173 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.759175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.759306 (+ 131us) server_negotiation.cc:407] Connection header received
0927 02:51:59.759404 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.759405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.759416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.759446 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.759831 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59816: BlockingRecv error: recv got EOF from 127.0.0.1:59816 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 02:51:59.760848 23315 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.759411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37268 (local address 127.0.98.60:34343)
0927 02:51:59.759494 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.759496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.760109 (+ 613us) server_negotiation.cc:407] Connection header received
0927 02:51:59.760200 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.760200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.760213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.760239 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.760789 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37268: BlockingRecv error: recv got EOF from 127.0.0.1:37268 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 02:51:59.767571 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.764498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.0.98.61:40583)
0927 02:51:59.764551 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.764553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.766987 (+ 2434us) server_negotiation.cc:407] Connection header received
0927 02:51:59.767093 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.767094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.767112 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.767155 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.767498 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59830: BlockingRecv error: recv got EOF from 127.0.0.1:59830 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 02:51:59.770787 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.770150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.0.98.61:40583)
0927 02:51:59.770181 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.770182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.770282 (+ 100us) server_negotiation.cc:407] Connection header received
0927 02:51:59.770369 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.770370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.770379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.770407 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.770737 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59846: BlockingRecv error: recv got EOF from 127.0.0.1:59846 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:51:59.775795 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.775058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.0.98.61:40583)
0927 02:51:59.775093 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.775094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.775300 (+ 206us) server_negotiation.cc:407] Connection header received
0927 02:51:59.775391 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.775392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.775402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.775429 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.775746 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59852: BlockingRecv error: recv got EOF from 127.0.0.1:59852 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 02:51:59.781957 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.781057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.0.98.61:40583)
0927 02:51:59.781093 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.781094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.781350 (+ 256us) server_negotiation.cc:407] Connection header received
0927 02:51:59.781475 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.781476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.781487 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.781516 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.781908 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59862: BlockingRecv error: recv got EOF from 127.0.0.1:59862 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 02:51:59.792053 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.791252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.98.61:40583)
0927 02:51:59.791293 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.791295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.791520 (+ 225us) server_negotiation.cc:407] Connection header received
0927 02:51:59.791627 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.791629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.791639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.791668 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.791995 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59878: BlockingRecv error: recv got EOF from 127.0.0.1:59878 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 02:51:59.809966 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.809311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.0.98.61:40583)
0927 02:51:59.809353 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.809355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.809485 (+ 130us) server_negotiation.cc:407] Connection header received
0927 02:51:59.809534 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.809535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.809546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.809591 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.809903 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59890: BlockingRecv error: recv got EOF from 127.0.0.1:59890 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 02:51:59.843529 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.842961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.0.98.61:40583)
0927 02:51:59.843017 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.843019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.843111 (+ 92us) server_negotiation.cc:407] Connection header received
0927 02:51:59.843162 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.843163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.843172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.843233 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.843482 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59896: BlockingRecv error: recv got EOF from 127.0.0.1:59896 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260927 02:51:59.909328 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:51:59.908683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.0.98.61:40583)
0927 02:51:59.908740 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 02:51:59.908742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:51:59.908791 (+ 49us) server_negotiation.cc:407] Connection header received
0927 02:51:59.908874 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:51:59.908875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:51:59.908885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:51:59.908929 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:51:59.909264 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59902: BlockingRecv error: recv got EOF from 127.0.0.1:59902 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 02:52:00.038959 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:52:00.038337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.0.98.61:40583)
0927 02:52:00.038402 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:00.038404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:00.038459 (+ 55us) server_negotiation.cc:407] Connection header received
0927 02:52:00.038523 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:00.038524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:00.038536 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:00.038580 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:00.038896 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59904: BlockingRecv error: recv got EOF from 127.0.0.1:59904 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 02:52:00.296741 23198 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 02:52:00.296166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59918 (local address 127.0.98.61:40583)
0927 02:52:00.296221 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:00.296223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:00.296279 (+ 56us) server_negotiation.cc:407] Connection header received
0927 02:52:00.296332 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:00.296332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:00.296342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:00.296390 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 02:52:00.296681 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59918: BlockingRecv error: recv got EOF from 127.0.0.1:59918 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 02:52:00.583106 23286 catalog_manager.cc:1259] Loaded cluster ID: 3ccd79debd764d83aa43291fdf0b0b2f
I20260927 02:52:00.583174 23286 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: loading cluster ID for follower catalog manager: success
I20260927 02:52:00.583926 23286 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: acquiring CA information for follower catalog manager: success
I20260927 02:52:00.584177 23286 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b29a861275a44622971f6aa080ee6679: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:52:00.659592 23330 catalog_manager.cc:1259] Loaded cluster ID: 3ccd79debd764d83aa43291fdf0b0b2f
I20260927 02:52:00.659672 23330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: loading cluster ID for follower catalog manager: success
I20260927 02:52:00.660477 23330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: acquiring CA information for follower catalog manager: success
I20260927 02:52:00.660778 23330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 13b8888f57ea4282b18789ede8285eaa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 02:52:00.812311 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:00.809914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.0.98.61:40583)
0927 02:52:00.810033 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:00.810034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:00.810046 (+ 12us) server_negotiation.cc:407] Connection header received
0927 02:52:00.810116 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:00.810116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:00.810168 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:00.810241 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:00.810360 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.810859 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.811587 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.811810 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.811858 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:00.812021 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:00.812135 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:00.812206 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:00.812219 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 02:52:00.812239 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260927 02:52:00.816071 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:00.813958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.0.98.60:34343)
0927 02:52:00.814048 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:00.814050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:00.814074 (+ 24us) server_negotiation.cc:407] Connection header received
0927 02:52:00.814137 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:00.814138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:00.814172 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:00.814228 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:00.814298 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.814759 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.815459 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.815652 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.815698 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:00.815823 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:00.815917 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:00.815968 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:00.815995 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 02:52:00.816010 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":52,"threads_started":1}
I20260927 02:52:00.819952 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:00.817845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.0.98.60:34343)
0927 02:52:00.817877 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:00.817879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:00.817885 (+ 6us) server_negotiation.cc:407] Connection header received
0927 02:52:00.817903 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:00.817904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:00.817912 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:00.817958 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:00.818089 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.818553 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.819311 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.819505 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.819564 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:00.819693 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:00.819729 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:00.819767 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:00.819857 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 02:52:00.819880 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:00.820535 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:00.817794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.0.98.61:40583)
0927 02:52:00.817830 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:00.817832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:00.817839 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:00.817861 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:00.817862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:00.817870 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:00.817912 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:00.818057 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.818765 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.819816 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.820078 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.820133 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:00.820306 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:00.820337 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:00.820402 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:00.820446 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 02:52:00.820467 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 02:52:00.820776 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 02:52:00.817577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.0.98.62:36337)
0927 02:52:00.817829 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 02:52:00.817832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 02:52:00.817839 (+ 7us) server_negotiation.cc:407] Connection header received
0927 02:52:00.817884 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 02:52:00.817885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 02:52:00.817931 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 02:52:00.818016 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 02:52:00.818141 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.818898 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.819839 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 02:52:00.820114 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 02:52:00.820171 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 02:52:00.820382 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 02:52:00.820479 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 02:52:00.820527 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 02:52:00.820679 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 02:52:00.820699 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
I20260927 02:52:00.822291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 23051
I20260927 02:52:00.827052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 23126
I20260927 02:52:00.831439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskc9ab2Y/build/release/bin/kudu with pid 23201
2026-09-27T02:52:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2729 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5436 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288697 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS